You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-08 - 20:11

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Sat Nov 08, 2025 12:43:59)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
23396210679,9sleep10-21swapper/107:08:231
2317829878,9sleep30-21swapper/307:05:333
2351728364,8sleep00-21swapper/007:09:550
2323228164,13sleep20-21swapper/207:06:152
207522670,0sleep10-21swapper/111:20:031
299242600,0sleep10-21swapper/112:28:261
127342470,0sleep20-21swapper/211:13:352
44292250,0sleep00-21swapper/012:12:360
2361899250,2cyclictest0-21swapper/312:10:153
2361699250,2cyclictest0-21swapper/207:25:152
2361699250,25cyclictest0-21swapper/211:25:112
23611992321,2cyclictest0-21swapper/012:05:320
2361899220,22cyclictest0-21swapper/310:34:403
2361899210,1cyclictest0-21swapper/310:04:253
2361699210,19cyclictest0-21swapper/210:25:572
23613992120,1cyclictest0-21swapper/112:16:071
2361199210,21cyclictest0-21swapper/011:57:140
23618992018,1cyclictest28020-21rm11:25:163
2361899200,0cyclictest0-21swapper/309:15:133
23616992020,0cyclictest0-21swapper/210:37:512
2361699200,20cyclictest0-21swapper/212:32:392
2361399200,18cyclictest0-21swapper/110:38:221
2361699194,4cyclictest0-21swapper/212:27:102
2361699180,17cyclictest0-21swapper/209:47:392
2361699176,9cyclictest0-21swapper/209:05:142
2361699171,11cyclictest0-21swapper/207:10:142
2361699170,17cyclictest0-21swapper/210:16:432
2361699170,16cyclictest0-21swapper/210:03:422
2361199173,6cyclictest16456-21diskmemload10:10:140
2361199170,17cyclictest0-21swapper/012:18:390
23616991611,4cyclictest18110-21grep12:20:272
2361699160,15cyclictest0-21swapper/212:36:392
2361699160,15cyclictest0-21swapper/211:25:002
2361199161,8cyclictest20641-21idleruntime-cro10:54:590
23611991615,1cyclictest0-21swapper/011:47:490
2361899153,11cyclictest1291-21runrttasks12:17:163
23616991513,2cyclictest0-21swapper/210:30:252
23616991510,3cyclictest1307-21mailstats08:35:232
2361399152,1cyclictest141rcu_preempt11:53:411
2361399151,4cyclictest224-21systemd-journal10:44:591
23613991512,2cyclictest25905-21cut08:20:171
23613991510,3cyclictest18091-21wc12:00:281
2361199150,14cyclictest0-21swapper/011:43:050
2361199150,14cyclictest0-21swapper/011:43:040
23618991413,1cyclictest39-21ksoftirqd/311:21:123
23618991410,3cyclictest32169-21cat08:35:013
2361899140,2cyclictest0-21swapper/312:05:083
2361899140,14cyclictest0-21swapper/309:44:423
2361699141,3cyclictest0-21swapper/211:15:132
23616991412,2cyclictest0-21swapper/211:00:162
23616991411,3cyclictest0-21swapper/211:30:142
2361699140,13cyclictest0-21swapper/210:53:232
23613991413,1cyclictest0-21swapper/110:15:141
23613991412,2cyclictest8772-21apache209:48:321
23613991410,3cyclictest25-21ksoftirqd/110:54:591
23613991410,3cyclictest18191-21grep08:05:151
23613991410,3cyclictest16468-21ls09:10:001
23613991410,3cyclictest16341-21sed10:25:201
2361399140,2cyclictest0-21swapper/112:34:591
2361399140,12cyclictest2573-21grep08:40:551
2361199145,6cyclictest1343-21runrttasks09:23:200
23611991413,1cyclictest0-21swapper/012:25:040
23611991410,3cyclictest9177-21cron10:20:010
23611991410,3cyclictest10285-21grep07:45:280
2361199140,13cyclictest0-21swapper/011:38:240
23618991310,2cyclictest6385-21grep08:45:283
23618991310,2cyclictest4071-21grep08:41:173
23618991310,2cyclictest3552-21fschecks_time09:25:163
23618991310,2cyclictest23207-21grep10:55:293
23618991310,2cyclictest19230-21uname08:05:223
23618991310,2cyclictest1596-21cat12:30:273
2361699139,2cyclictest20569-21cut08:10:162
2361699139,2cyclictest12756-21tail09:00:202
2361699137,5cyclictest11325-21ntp_states11:35:222
2361699134,4cyclictest224-21systemd-journal07:49:592
23616991312,1cyclictest0-21swapper/207:31:082
2361699131,1cyclictest10715-21ssh12:15:572
23616991310,2cyclictest9105-21grep11:55:192
23616991310,2cyclictest6418-21grep07:40:162
23616991310,2cyclictest29920-21grep10:10:152
23616991310,2cyclictest27380-21cron08:25:012
23616991310,2cyclictest2435-21grep10:40:182
2361399139,3cyclictest5175-21wc07:35:251
2361399139,2cyclictest32179-21wc07:25:211
2361399139,2cyclictest18587-21sed10:00:281
2361399131,12cyclictest16456-21diskmemload09:14:241
23613991310,2cyclictest5243-21sh11:07:081
23613991310,2cyclictest28352-21munin-run07:19:591
23613991310,2cyclictest11879-21grep11:35:271
2361199139,3cyclictest8679-21grep10:45:170
2361199139,3cyclictest17400-21sed10:25:290
2361199135,7cyclictest20491-21munin-run11:20:000
2361199133,8cyclictest0-21swapper/011:04:530
23611991310,2cyclictest8451-21tr12:35:160
23611991310,2cyclictest23664-21grep08:15:180
23611991310,2cyclictest16480-21basename12:00:150
23611991310,2cyclictest1321-21tr12:30:230
2361199130,12cyclictest22142-21head10:55:200
2361199130,0cyclictest0-21swapper/010:38:230
2361899129,2cyclictest9024-21sed10:45:183
2361899129,2cyclictest3652-21expr09:50:163
2361899129,2cyclictest30803-21sort09:20:263
2361899129,2cyclictest27855-21cat10:35:173
2361899129,2cyclictest25959-21munin-plugin-st07:15:003
2361899129,2cyclictest16826-21grep10:25:243
2361899129,2cyclictest16685-21basename09:35:183
2361899129,2cyclictest16447-21grep08:00:183
2361899129,2cyclictest16060-21fgrep11:15:233
2361899129,2cyclictest10854-21ssh11:11:123
2361899129,2cyclictest10699-21dump-pmu-power07:50:003
2361899129,2cyclictest1058-21head11:50:193
2361899128,3cyclictest15968-21grep09:05:293
2361899124,7cyclictest0-21swapper/308:20:143
2361899122,9cyclictest6641-21df_abs07:40:163
23618991211,1cyclictest0-21swapper/310:43:003
23618991210,1cyclictest28732-21tr07:20:143
23618991210,1cyclictest13499-21ssh09:56:573
2361699129,2cyclictest8316-21tr10:45:162
2361699129,2cyclictest7710-21sleep11:55:012
2361699129,2cyclictest723-21tail12:10:192
2361699129,2cyclictest4484-21tail07:35:172
2361699129,2cyclictest24271-21grep09:15:222
2361699129,2cyclictest24041-21grep10:05:172
2361699129,2cyclictest23154-21grep08:15:162
2361699127,4cyclictest26519-21sed11:45:262
2361699122,10cyclictest61850irq/126-enp2s0-12:02:462
2361699120,2cyclictest20258-21kworker/2:011:06:022
2361399129,2cyclictest9784-21tr10:45:381
2361399129,2cyclictest706-21grep08:35:191
2361399129,2cyclictest6292-21grep07:40:151
2361399129,2cyclictest25534-21grep12:05:201
2361399129,2cyclictest24592-21sed07:10:181
2361399129,2cyclictest19087-21sed11:40:281
2361399129,2cyclictest19087-21sed11:40:271
2361399129,2cyclictest18027-21sed12:20:271
2361399129,2cyclictest16418-21sed11:15:281
2361399129,2cyclictest14848-21taskset09:58:241
2361399129,2cyclictest12748-21cut09:00:211
2361399128,3cyclictest28667-21ssh09:20:001
2361399128,3cyclictest11194-21grep07:50:181
2361399126,5cyclictest8761-21apache209:34:131
2361399124,4cyclictest141rcu_preempt11:55:161
23613991210,1cyclictest24487-21ntpq10:05:211
2361399120,2cyclictest0-21swapper/111:00:301
2361199129,2cyclictest9227-21head11:10:210
2361199129,2cyclictest885-21cron07:29:590
2361199129,2cyclictest7373-21grep07:40:230
2361199129,2cyclictest5431-21grep07:35:260
2361199129,2cyclictest5145-21grep09:50:280
2361199129,2cyclictest29778-21ls08:30:010
2361199129,2cyclictest25100-21sed10:05:260
2361199129,2cyclictest24794-21sed09:15:270
2361199129,2cyclictest23693-21munin-run12:25:000
2361199129,2cyclictest21247-21cut11:20:160
2361199129,2cyclictest17651-21uname09:35:240
2361199129,2cyclictest13072-21sh07:55:000
2361199128,3cyclictest6859-21cron08:49:590
2361199128,2cyclictest21101-21sort08:10:200
2361199120,11cyclictest18879-21tail08:05:180
2361199120,11cyclictest0-21swapper/011:05:280
145292120,2sleep314530-21sh10:24:383
2361899119,1cyclictest17184-21cut12:00:193
2361899118,2cyclictest8423-21wc11:55:163
2361899118,2cyclictest7053-21ssh11:32:383
2361899118,2cyclictest6573-21ssh11:08:353
2361899118,2cyclictest5217-21cat07:35:263
2361899118,2cyclictest31572-21ssh11:02:173
2361899118,2cyclictest27738-21taskset11:46:243
2361899118,2cyclictest24584-21grep08:15:283
2361899118,2cyclictest22522-21awk09:15:003
2361899118,2cyclictest21851-21cut08:10:253
2361899118,2cyclictest15437-21taskset07:59:563
2361899118,2cyclictest14884-21expr10:50:163
2361899118,2cyclictest14312-21ssh11:37:333
2361899118,2cyclictest12278-21ssh09:30:583
2361899110,10cyclictest25095-21sed10:05:273
2361899110,10cyclictest13797-21taskset09:01:313
2361699119,1cyclictest30649-21cut08:30:182
2361699119,1cyclictest10437-21tail10:20:192
2361699118,2cyclictest94-21kswapd009:50:132
2361699118,2cyclictest8941-21taskset08:50:322
2361699118,2cyclictest6964-21chrt09:27:482
2361699118,2cyclictest4755-21apache_processe08:45:132
2361699118,2cyclictest3117-21sort08:41:012
2361699118,2cyclictest30442-21ssh09:20:222
2361699118,2cyclictest27853-21sed07:15:252
2361699118,2cyclictest25920-21ssh12:05:252
2361699118,2cyclictest23151-21grep10:55:302
2361699118,2cyclictest20931-21taskset11:42:172
2361699118,2cyclictest20931-21taskset11:42:162
2361699118,2cyclictest17140-21tr09:10:152
2361699118,2cyclictest16788-21grep09:35:182
2361699118,2cyclictest14107-21expr07:55:202
2361699118,2cyclictest11789-21ssh09:55:262
2361699118,2cyclictest11788-21cat09:00:002
2361699118,2cyclictest11000-21grep07:50:172
2361699118,2cyclictest10036-21egrep09:30:162
2361699117,1cyclictest0-21swapper/209:41:252
2361699111,9cyclictest21063-21munin-node08:04:352
2361399118,2cyclictest9943-21cut10:20:161
2361399118,2cyclictest8096-21ssh12:35:121
2361399118,2cyclictest5637-21ssh09:50:511
2361399118,2cyclictest3720-21sh12:12:011
2361399118,2cyclictest3676-21grep09:25:181
2361399118,2cyclictest28596-21tr08:25:231
2361399118,2cyclictest28033-21apt11:25:161
2361399118,2cyclictest26702-21ssh09:42:401
2361399118,2cyclictest25-21ksoftirqd/111:45:171
2361399118,2cyclictest22459-21grep10:30:181
2361399118,2cyclictest22325-21sh10:55:221
2361399118,2cyclictest2130-21ssh10:14:001
2361399118,2cyclictest16486-21grep08:00:181
2361399118,2cyclictest14829-21wc07:55:281
2361399118,2cyclictest10244-21cut08:55:181
2361399117,3cyclictest4634-21wc11:30:271
2361399115,2cyclictest224-21systemd-journal08:18:261
2361199119,1cyclictest9857-21grep08:55:160
2361199119,1cyclictest7616-21df_abs08:50:160
2361199118,2cyclictest9660-21taskset09:30:060
2361199118,2cyclictest7007-21ssh11:54:260
2361199118,2cyclictest565-21ssh09:47:260
2361199118,2cyclictest4863-21ssh11:30:300
2361199118,2cyclictest4285-21seq08:42:290
2361199118,2cyclictest3280-21turbostat.cron07:35:000
2361199118,2cyclictest30627-21ssh11:26:040
2361199118,2cyclictest2841-21cat09:25:120
2361199118,2cyclictest23873-21sed09:40:180
2361199118,2cyclictest22383-21ssh09:14:520
2361199118,2cyclictest21219-21ssh10:03:130
2361199118,2cyclictest2024-21expr10:40:160
2361199118,2cyclictest13973-21sort07:55:190
2361199118,2cyclictest13039-21ssh10:22:550
2361199117,3cyclictest27420-21ls07:15:200
2361199117,3cyclictest17298-21sed08:00:270
2361899108,1cyclictest7117-21basename08:50:113
2361899108,1cyclictest4034-21expr10:15:163
2361899108,1cyclictest32027-21ssh09:46:223
2361899108,1cyclictest31549-21cat07:25:173
2361899108,1cyclictest24975-21cut12:25:203
2361899108,1cyclictest12172-21cut07:50:253
2361899108,1cyclictest10999-21ssh12:36:033
2361899107,2cyclictest29586-21ssh10:10:093
2361899107,2cyclictest17017-21head12:20:173
2361899100,9cyclictest17150-21ssh11:40:103
2361899100,9cyclictest17150-21ssh11:40:103
2361899100,6cyclictest1883-21sshd08:37:253
2361699108,1cyclictest305902sleep207:23:192
2361699108,1cyclictest29683-21taskset08:29:062
2361699107,2cyclictest19086-21sort08:05:212
2361399108,1cyclictest9729-21ntpq07:45:231
2361399108,1cyclictest5862-21head08:45:211
2361399107,2cyclictest30033-21taskset08:30:141
2361399107,2cyclictest30005-21basename09:20:171
2361399107,2cyclictest2439-21wc07:30:271
2361399107,2cyclictest21436-21sed08:10:231
2361399107,2cyclictest11427-21sh11:11:501
2361399106,3cyclictest17335-21cut09:35:221
2361199108,1cyclictest14628-21expr09:05:160
2361199107,2cyclictest30708-21chrt07:24:490
2361199107,2cyclictest30333-21cat08:30:150
2361199107,2cyclictest25416-21cut08:20:140
2361199107,2cyclictest24626-21grep07:10:180
2361199107,2cyclictest21676-21apache_accesses10:30:130
2361199107,2cyclictest13123-21ntp_states09:00:250
2361199107,2cyclictest10610-21cat09:55:150
236189998,1cyclictest2205-21grep07:30:233
236189996,2cyclictest28133-21seq07:17:323
236139997,1cyclictest8628-21cat08:50:271
236139997,1cyclictest29481-21mii-tool07:20:181
236119995,3cyclictest510-21grep08:35:170
236189986,1cyclictest9550-21basename08:55:143
236189986,1cyclictest27690-21wget08:25:153
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional