You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-01 - 02:55

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Wed Apr 01, 2026 00:44:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
109932202165,24sleep30-21swapper/319:07:053
110572201164,25sleep00-21swapper/019:07:540
109922201164,24sleep20-21swapper/219:07:042
109212201164,25sleep10-21swapper/119:06:101
1881221350,4sleep01131399cyclictest00:20:230
11323991070,106cyclictest0-21swapper/200:11:472
113239910299,2cyclictest0-21swapper/223:04:572
11323997874,3cyclictest0-21swapper/222:19:372
205322640,1sleep10-21swapper/123:10:011
320582510,2sleep00-21swapper/022:20:200
11323995137,3cyclictest0-21swapper/221:54:262
256842490,4sleep10-21swapper/122:05:241
11323994845,2cyclictest0-21swapper/221:40:032
11323994833,5cyclictest0-21swapper/221:27:242
11330994529,4cyclictest0-21swapper/300:06:243
152972430,1sleep1221ktimersoftd/119:15:171
11323994231,3cyclictest0-21swapper/222:55:262
11323994226,11cyclictest0-21swapper/219:38:342
116762410,1sleep00-21swapper/000:00:240
11323994128,3cyclictest0-21swapper/220:56:362
11323994126,9cyclictest0-21swapper/221:37:202
11330994031,3cyclictest0-21swapper/321:54:263
11330994029,3cyclictest0-21swapper/300:24:163
11323994027,8cyclictest0-21swapper/222:20:462
11330993926,7cyclictest0-21swapper/321:27:243
11323993929,3cyclictest0-21swapper/220:31:102
1132399391,31cyclictest0-21swapper/221:45:522
1131899389,5cyclictest19064-21smart_sda21:50:231
1131899381,11cyclictest24231-21cat22:05:001
1131399389,13cyclictest32365-21awk21:10:000
1132399371,15cyclictest0-21swapper/222:42:502
11318993712,3cyclictest19674-21/usr/sbin/munin00:25:261
11330993622,3cyclictest0-21swapper/320:34:583
11323993627,3cyclictest0-21swapper/222:07:322
11323993615,13cyclictest0-21swapper/222:04:142
1132399361,11cyclictest0-21swapper/223:35:062
1132399361,11cyclictest0-21swapper/223:35:062
1132399361,11cyclictest0-21swapper/221:34:242
11318993611,3cyclictest2422-21/usr/sbin/munin21:15:231
1131899361,11cyclictest4997-21sh22:35:001
1131399366,14cyclictest15382-21munin-run21:45:000
11330993527,6cyclictest0-21swapper/300:15:283
11330993526,2cyclictest0-21swapper/322:15:223
11330993521,3cyclictest0-21swapper/320:13:103
11330993517,10cyclictest0-21swapper/323:38:403
11330993517,10cyclictest0-21swapper/323:38:403
11323993521,3cyclictest4316-21hddtemp_smartct23:45:142
1132399351,9cyclictest0-21swapper/200:06:382
1132399351,32cyclictest0-21swapper/200:24:162
1132399351,28cyclictest0-21swapper/200:19:202
11318993513,19cyclictest10031-40updatedb00:00:001
1131899351,16cyclictest0-21swapper/100:04:301
1131899351,12cyclictest16813-21grep23:00:161
1131899351,11cyclictest10427-21cat20:20:001
1131899351,10cyclictest456-21df21:10:131
1131899350,15cyclictest0-21swapper/122:56:441
11323993421,8cyclictest0-21swapper/219:51:202
11323993420,3cyclictest0-21swapper/221:55:362
11323993419,9cyclictest0-21swapper/219:23:102
1132399341,31cyclictest0-21swapper/219:26:422
11318993411,3cyclictest14032-21gltestperf21:40:131
1131899341,10cyclictest27368-21grep23:25:121
11330993325,3cyclictest0-21swapper/300:34:143
11330993323,8cyclictest0-21swapper/322:55:263
11330993317,10cyclictest0-21swapper/320:28:463
1133099330,25cyclictest0-21swapper/322:09:243
1133099330,24cyclictest0-21swapper/322:24:153
1132399337,3cyclictest0-21swapper/220:37:442
11323993329,3cyclictest0-21swapper/220:13:102
11323993328,3cyclictest0-21swapper/223:26:042
11323993322,3cyclictest0-21swapper/221:08:322
11323993317,8cyclictest0-21swapper/220:00:162
11323993317,8cyclictest0-21swapper/220:00:162
1131899339,4cyclictest12582-21/usr/sbin/munin20:25:001
1131899339,3cyclictest13625-21munin-node00:20:101
1131899338,4cyclictest1-21systemd21:30:001
1131899331,6cyclictest405-1kworker/u9:000:15:171
1131899331,2cyclictest0-21swapper/121:36:401
1131899331,16cyclictest0-21swapper/121:09:041
1131899331,10cyclictest29442-21smartctl21:00:261
11318993310,3cyclictest27007-21ntp_kernel_err20:55:191
115242320,3sleep11131899cyclictest19:10:121
11330993223,3cyclictest0-21swapper/321:17:123
11330993220,3cyclictest0-21swapper/323:47:043
11330993220,3cyclictest0-21swapper/322:11:283
1133099321,14cyclictest0-21swapper/300:37:153
1133099321,13cyclictest7763-21df_inode22:40:113
11323993228,3cyclictest0-21swapper/222:11:282
11323993219,2cyclictest0-21swapper/219:55:262
11323993217,10cyclictest0-21swapper/223:32:222
1132399321,29cyclictest0-21swapper/221:17:122
1132399321,18cyclictest0-21swapper/223:16:282
1131899329,3cyclictest29407-21/usr/sbin/munin23:30:131
1131899329,3cyclictest16595-21/usr/sbin/munin19:20:171
1131899328,21cyclictest1177-21who19:55:271
1131899325,3cyclictest0-21swapper/120:50:211
1131899324,25cyclictest7601-21cstates22:40:111
1131899321,4cyclictest6590-21munin-run21:25:001
11318993212,2cyclictest0-21swapper/122:49:421
1131899321,12cyclictest19210-21kworker/u8:023:17:021
1131899321,10cyclictest0-21swapper/123:14:561
11318993210,3cyclictest27566-21/usr/sbin/munin19:45:131
11318993210,3cyclictest15800-21interrupts20:30:161
1131899320,1cyclictest0-21swapper/123:51:461
1131899320,14cyclictest0-21swapper/119:37:361
11330993116,4cyclictest31448-21ntp_states21:05:183
11330993116,4cyclictest16667-21ntp_states21:45:203
1133099311,23cyclictest0-21swapper/319:33:513
1132399317,3cyclictest0-21swapper/220:40:382
11323993127,3cyclictest0-21swapper/219:43:402
11323993120,3cyclictest0-21swapper/220:52:202
11323993117,7cyclictest0-21swapper/222:50:382
11323993116,3cyclictest0-21swapper/220:27:062
1132399311,19cyclictest0-21swapper/223:21:582
1132399311,18cyclictest0-21swapper/222:45:362
1132399311,15cyclictest0-21swapper/220:16:022
1131899319,4cyclictest0-21swapper/120:10:131
1131899319,3cyclictest12741-21missed_timers22:50:181
11318993113,15cyclictest28794-21/usr/sbin/munin22:15:111
11318993112,2cyclictest0-21swapper/122:26:141
11318993112,2cyclictest0-21swapper/122:26:141
1131899311,12cyclictest19210-21kworker/u8:000:34:131
1131399311,15cyclictest1211-21awk23:40:000
1131399311,15cyclictest1211-21awk23:40:000
11330993019,3cyclictest0-21swapper/320:15:483
11330993018,3cyclictest0-21swapper/321:40:263
1133099301,14cyclictest0-21swapper/321:22:003
1133099301,13cyclictest0-21swapper/321:30:153
1132399307,3cyclictest0-21swapper/219:12:522
1132399306,3cyclictest0-21swapper/220:46:202
11323993016,7cyclictest0-21swapper/223:10:282
1132399301,19cyclictest0-21swapper/223:52:582
1132399301,17cyclictest0-21swapper/200:30:322
1132399301,11cyclictest0-21swapper/200:37:042
1131899309,3cyclictest827-21runrttasks23:20:531
1131899308,2cyclictest3658-21grep23:45:101
1131899301,8cyclictest808-21snmpd21:32:091
1131899301,8cyclictest12613-21sa200:07:101
11318993010,3cyclictest827-21runrttasks00:10:521
11318993010,3cyclictest30006-21ls19:50:111
1131899300,14cyclictest0-21swapper/123:39:131
1131899300,14cyclictest0-21swapper/123:39:131
1131899300,14cyclictest0-21swapper/122:11:401
11330992916,3cyclictest0-21swapper/321:57:023
11330992913,3cyclictest0-21swapper/319:26:163
11330992912,9cyclictest0-21swapper/319:58:003
11330992912,3cyclictest0-21swapper/320:59:563
1133099291,12cyclictest0-21swapper/323:33:403
1133099291,12cyclictest0-21swapper/319:41:223
1133099291,11cyclictest0-21swapper/322:52:183
1132399291,9cyclictest0-21swapper/221:12:062
11323992917,4cyclictest0-21swapper/220:08:242
1132399291,16cyclictest0-21swapper/222:38:162
1132399291,12cyclictest0-21swapper/222:32:582
1132399291,12cyclictest0-21swapper/220:22:262
1131899299,3cyclictest32574-21uptime22:20:251
1131899298,3cyclictest26030-21latency_hist00:40:001
1131899292,3cyclictest6035-21irqstats22:35:161
1131899291,6cyclictest0-21swapper/123:40:131
1131899291,5cyclictest19999-21/usr/sbin/munin21:55:261
1131899291,3cyclictest23609-21cat20:50:001
1131899291,11cyclictest2867-21ntp_states20:00:191
1131899291,11cyclictest2867-21ntp_states20:00:191
1131899291,10cyclictest0-21swapper/119:40:181
11330992822,3cyclictest0-21swapper/320:24:023
11330992817,3cyclictest0-21swapper/322:26:263
11330992817,3cyclictest0-21swapper/322:26:263
11330992814,10cyclictest0-21swapper/319:16:483
1133099280,15cyclictest0-21swapper/322:02:073
1132399286,7cyclictest0-21swapper/219:34:162
1132399281,9cyclictest0-21swapper/223:57:382
11323992814,9cyclictest0-21swapper/221:21:162
11323992814,7cyclictest0-21swapper/219:18:102
1132399281,16cyclictest0-21swapper/200:29:502
1132399281,13cyclictest0-21swapper/200:01:422
1131899281,9cyclictest0-21swapper/120:38:191
1131899281,4cyclictest22957-21awk19:35:001
11318992810,2cyclictest21369-21awk20:45:011
1131899280,16cyclictest0-21swapper/121:48:421
11330992722,3cyclictest0-21swapper/323:26:043
11330992711,9cyclictest0-21swapper/319:11:303
1133099271,13cyclictest0-21swapper/320:40:233
1133099271,11cyclictest0-21swapper/323:40:123
11323992717,3cyclictest0-21swapper/221:00:102
1131899271,15cyclictest0-21swapper/120:27:341
1131399272,12cyclictest0-21swapper/020:25:000
11330992616,3cyclictest0-21swapper/319:45:103
11330992615,3cyclictest0-21swapper/319:35:003
11330992613,8cyclictest0-21swapper/321:12:423
11330992612,2cyclictest0-21swapper/300:03:363
11323992611,3cyclictest0-21swapper/222:26:402
11323992611,3cyclictest0-21swapper/222:26:402
11323992611,3cyclictest0-21swapper/219:47:222
1131899263,3cyclictest87550irq/128-enp1s0-19:29:361
1131899261,10cyclictest0-21swapper/120:05:541
1131399261,10cyclictest32427-21grep19:55:160
11313992610,3cyclictest17861-21kworker/0:023:15:190
11313992610,14cyclictest26385-21awk22:10:000
1131399260,12cyclictest7191-21idleruntime-cro22:40:000
1133099255,3cyclictest0-21swapper/300:28:303
11330992516,3cyclictest0-21swapper/320:06:403
1133099251,14cyclictest0-21swapper/320:49:133
1133099251,13cyclictest0-21swapper/323:10:263
11330992510,10cyclictest0-21swapper/322:34:483
1133099250,11cyclictest0-21swapper/300:13:113
1132399257,3cyclictest0-21swapper/223:07:142
11323992512,3cyclictest0-21swapper/223:41:062
1131399252,12cyclictest3787-21munin-run20:05:000
1131399252,12cyclictest3787-21munin-run20:05:000
1131399251,6cyclictest0-21swapper/000:37:320
11330992413,2cyclictest0-21swapper/323:55:003
1133099241,17cyclictest0-21swapper/321:35:133
1133099241,14cyclictest0-21swapper/319:54:383
1133099241,13cyclictest0-21swapper/323:57:273
11330992411,3cyclictest0-21swapper/319:20:303
1133099241,10cyclictest0-21swapper/323:24:463
1133099241,10cyclictest0-21swapper/323:08:393
11330992410,3cyclictest0-21swapper/322:48:503
1133099240,14cyclictest0-21swapper/321:03:343
87002230,1sleep00-21swapper/020:15:120
1133099237,3cyclictest0-21swapper/320:50:463
1133099231,13cyclictest0-21swapper/320:02:363
1133099231,13cyclictest0-21swapper/320:02:363
1133099231,11cyclictest0-21swapper/320:35:233
1131399231,5cyclictest0-21swapper/023:55:370
1131399227,14cyclictest0-21swapper/000:10:010
1131399221,4cyclictest12005-21kworker/u8:122:57:210
1131399219,3cyclictest27121-21grep20:55:220
1131399215,8cyclictest0-21swapper/019:49:260
1131399212,11cyclictest759-21/usr/sbin/munin22:25:240
1131399212,11cyclictest759-21/usr/sbin/munin22:25:240
1131399211,2cyclictest0-21swapper/000:10:090
1131399211,10cyclictest0-21swapper/023:00:150
1133099209,3cyclictest0-21swapper/323:01:203
1133099201,14cyclictest23397-21perl23:15:133
1131399203,14cyclictest5284-21cut21:20:150
1131399202,5cyclictest990-1kworker/u9:121:32:280
1131399202,10cyclictest25373-21/usr/sbin/munin19:40:140
1131399201,9cyclictest0-21swapper/021:35:130
1131399201,5cyclictest405-1kworker/u9:000:26:070
1131399201,4cyclictest19071-21kworker/u8:121:49:280
1131399201,11cyclictest3641-21/usr/sbin/munin23:45:230
1131399201,11cyclictest20265-21sshd23:08:260
1131399201,10cyclictest6828-21head23:50:170
1131399201,10cyclictest23012-21grep00:30:210
1131399201,10cyclictest20623-21/usr/sbin/munin23:10:150
11313992010,6cyclictest0-21swapper/022:14:550
1131399200,4cyclictest10698-21kworker/u8:221:53:000
1131399198,3cyclictest6813-21/usr/sbin/munin21:25:130
1131399198,3cyclictest11824-21/usr/sbin/munin22:50:140
1131399196,4cyclictest15370-21sed19:15:180
1131399193,9cyclictest11-21rcuc/020:10:230
1131399193,13cyclictest20163-21tr21:55:130
1131399192,13cyclictest19830-21tune2fs20:40:130
1131399191,7cyclictest29861-21ntp_states22:15:180
1131399191,5cyclictest10879-21smart_sda22:45:230
1131399191,10cyclictest766-21ntpd22:00:110
1133099186,3cyclictest0-21swapper/322:38:483
1131399187,3cyclictest31057-21switchtime19:50:230
1131399187,2cyclictest24929-21pmu-power20:50:210
1131399182,6cyclictest0-21swapper/022:44:320
1131399181,8cyclictest0-21swapper/022:33:080
1131399181,7cyclictest17589-21cut19:20:200
1131399181,5cyclictest0-21swapper/023:25:140
1131399181,5cyclictest0-21swapper/000:18:550
11313991812,4cyclictest30026-21grep23:30:140
1131399181,10cyclictest25015-21/usr/sbin/munin23:20:190
1131399181,10cyclictest22657-21sshd19:31:210
1131399181,10cyclictest11-21rcuc/020:45:220
1131399181,10cyclictest0-21swapper/023:43:480
1131399178,2cyclictest87550irq/128-enp1s0-21:11:270
1131399178,2cyclictest23181-21/usr/sbin/munin19:35:160
1131399176,3cyclictest87550irq/128-enp1s0-20:25:160
1131399171,8cyclictest808-21snmpd19:25:340
1131399171,7cyclictest0-21swapper/021:17:310
1131399171,4cyclictest808-21snmpd20:34:490
1131399171,13cyclictest4167-21cut20:05:130
1131399164,3cyclictest87550irq/128-enp1s0-21:00:260
11313991611,3cyclictest12810-21sed19:10:240
1131399151,6cyclictest17193-21/usr/sbin/munin20:35:100
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional