You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-08 - 07:20

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Sat Aug 08, 2026 00:44:27)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
17712234164,57sleep20-21swapper/219:07:362
17752204166,25sleep10-21swapper/119:07:391
16322203167,24sleep30-21swapper/319:05:533
16342202166,24sleep00-21swapper/019:05:530
1462321740,3sleep3206399cyclictest00:30:153
2057991081,2cyclictest0-21swapper/221:34:382
2057998884,3cyclictest0-21swapper/220:47:112
221162480,3sleep3206399cyclictest23:34:083
205299438,4cyclictest244682open_files21:10:211
293332420,4sleep3401ktimersoftd/320:10:013
261802420,1sleep10-21swapper/122:30:011
205299423,6cyclictest18639-40dnf22:10:251
2052993814,21cyclictest19402-21/usr/sbin/munin19:45:211
205299381,23cyclictest0-21swapper/119:29:251
205299381,11cyclictest3828-21/usr/sbin/munin19:10:251
205299372,3cyclictest808-21snmpd23:35:081
2052993715,18cyclictest10170-21grep21:50:251
205299371,12cyclictest25357-21kworker/u8:020:16:481
205299371,10cyclictest14507-21awk22:00:261
205799361,28cyclictest0-21swapper/221:36:442
205299359,3cyclictest14947-21hddtemp_smartct00:30:171
2052993513,19cyclictest9645-21sh00:20:011
2052993512,20cyclictest16470-21/usr/sbin/munin00:35:261
205299351,12cyclictest4452-21processes00:05:211
205299351,12cyclictest24961-21df_inode23:40:131
2057993422,3cyclictest0-21swapper/221:50:272
2057993417,3cyclictest0-21swapper/219:49:272
205299348,4cyclictest23-21ksoftirqd/119:50:371
205299343,6cyclictest11781-21diskstats23:10:131
2052993414,17cyclictest9959-21latency_hist20:40:011
2052993412,3cyclictest12384-21/usr/sbin/munin20:45:191
2052993411,2cyclictest1375-21/usr/sbin/munin20:20:241
205299341,11cyclictest2657-21users00:00:271
205799331,16cyclictest0-21swapper/222:35:312
205299339,3cyclictest4559-21df_inode21:40:131
205299339,3cyclictest0-21swapper/122:21:251
205299331,3cyclictest739-21sh00:00:011
2052993313,17cyclictest28235-21ntp_states20:05:221
2052993313,17cyclictest28235-21ntp_states20:05:221
205299331,13cyclictest17154-21needreboot19:40:161
2052993311,3cyclictest1511-21seq21:31:531
205299331,10cyclictest0-21swapper/123:20:581
205299331,10cyclictest0-21swapper/123:02:431
2052993310,3cyclictest27238-21ntp_kernel_pll_22:30:191
2052993310,3cyclictest13971-21/usr/sbin/munin19:35:221
204799335,13cyclictest28185-21awk22:35:010
205799321,25cyclictest0-21swapper/222:03:032
205799321,15cyclictest0-21swapper/223:45:042
205799321,15cyclictest0-21swapper/223:45:032
205799321,14cyclictest0-21swapper/222:40:472
205299328,3cyclictest3602-21/usr/sbin/munin20:25:141
205299327,22cyclictest26976-21expr23:45:151
205299327,22cyclictest26976-21expr23:45:151
205299325,4cyclictest0-21swapper/119:15:191
2057993117,8cyclictest0-21swapper/221:28:152
205799311,19cyclictest0-21swapper/223:20:072
205799311,12cyclictest0-21swapper/222:59:512
205299314,9cyclictest755-21rngd00:21:071
205299311,3cyclictest1003-21head22:45:151
205299311,2cyclictest11448-21smart_sda20:40:231
205299311,11cyclictest26652-21ntp_states21:15:211
2052993110,3cyclictest16628-21cat20:55:011
204799314,12cyclictest5275-21idleruntime-cro00:10:010
205799301,12cyclictest26016-21ntp_states20:00:192
205299302,5cyclictest8620-21smart_sda19:20:231
2052993011,3cyclictest4859-21cpu22:55:111
205299301,13cyclictest31048-21ntp_states21:25:211
205299301,10cyclictest30029-21kworker/u8:023:28:071
2052993010,3cyclictest827-21runrttasks23:33:311
2052993010,3cyclictest29014-21/usr/sbin/munin23:50:221
2052993010,3cyclictest21583-21cut21:05:151
2052993010,2cyclictest7907-21unixbench_multi21:45:261
2052993010,2cyclictest17856-21open_files20:55:231
204799301,14cyclictest1792-21munin-run21:35:010
205799291,19cyclictest0-21swapper/223:29:112
205799291,19cyclictest0-21swapper/200:20:512
205799291,18cyclictest0-21swapper/222:21:512
205799291,15cyclictest0-21swapper/200:14:152
205799291,12cyclictest0-21swapper/223:43:232
205799291,12cyclictest0-21swapper/222:11:352
205799291,11cyclictest0-21swapper/221:15:192
205799290,17cyclictest0-21swapper/222:51:372
205799290,13cyclictest0-21swapper/222:09:472
205299299,3cyclictest29191-21cat22:35:161
205299296,2cyclictest7448-21latency_hist00:15:011
205299292,5cyclictest30624-21smart_sda20:10:231
205299291,3cyclictest20674-21ntp_states22:15:221
205299291,26cyclictest0-21swapper/121:57:461
2052992911,3cyclictest2981-21sh22:50:131
205299291,12cyclictest0-21swapper/120:33:211
205299291,12cyclictest0-21swapper/119:32:051
204799295,11cyclictest761-21awk00:00:010
204799294,12cyclictest0-21swapper/019:40:010
204799290,14cyclictest18763-21idleruntime-cro21:00:010
205799281,14cyclictest0-21swapper/222:18:352
205799281,13cyclictest0-21swapper/221:57:072
205799281,13cyclictest0-21swapper/221:42:112
205799281,13cyclictest0-21swapper/220:06:312
205799281,13cyclictest0-21swapper/220:06:312
205799281,13cyclictest0-21swapper/219:22:412
205799281,13cyclictest0-21swapper/200:38:512
205799281,13cyclictest0-21swapper/200:18:032
205799281,12cyclictest0-21swapper/221:45:432
205799281,12cyclictest0-21swapper/221:20:512
205799281,12cyclictest0-21swapper/220:37:172
205799281,11cyclictest0-21swapper/220:44:032
205799281,11cyclictest0-21swapper/200:00:552
205799280,12cyclictest0-21swapper/222:32:132
205799280,11cyclictest0-21swapper/220:17:212
205299288,3cyclictest2777-21cat21:35:171
205299288,3cyclictest19173-21expr21:00:141
205299285,3cyclictest8564-1kworker/u9:121:22:041
205299281,9cyclictest26661-21taskset20:01:371
205299281,3cyclictest32653-21cat22:45:011
205299281,3cyclictest24775-21cat20:00:011
205299281,25cyclictest0-21swapper/123:18:461
2047992810,15cyclictest16567-21awk20:55:000
205799278,9cyclictest0-21swapper/200:30:152
2057992718,3cyclictest0-21swapper/219:55:232
2057992711,4cyclictest0-21swapper/220:50:152
205799271,12cyclictest0-21swapper/223:37:172
205799271,12cyclictest0-21swapper/222:27:032
205799271,12cyclictest0-21swapper/219:13:192
205799271,11cyclictest0-21swapper/200:29:472
205299271,24cyclictest0-21swapper/122:06:431
2052992710,3cyclictest13207-21head00:25:211
205799261,12cyclictest0-21swapper/223:11:032
205799261,12cyclictest0-21swapper/219:16:152
205799261,11cyclictest0-21swapper/222:45:072
205799261,11cyclictest0-21swapper/220:20:092
2057992610,8cyclictest0-21swapper/200:08:352
205799260,9cyclictest0-21swapper/223:05:192
205799260,11cyclictest0-21swapper/221:05:092
205799260,11cyclictest0-21swapper/219:35:232
205799260,10cyclictest0-21swapper/221:04:492
205799260,10cyclictest0-21swapper/219:51:352
205299261,2cyclictest0-21swapper/123:07:391
204799261,7cyclictest0-21swapper/023:10:570
204799261,13cyclictest20975-21idleruntime-cro21:05:010
253272250,4sleep011-21rcuc/022:25:230
206399251,11cyclictest808-21snmpd00:35:253
205799255,3cyclictest0-21swapper/223:59:332
205799251,9cyclictest0-21swapper/223:30:312
205799251,9cyclictest0-21swapper/220:10:412
205799251,11cyclictest0-21swapper/221:14:272
205799251,11cyclictest0-21swapper/219:34:112
205799250,9cyclictest0-21swapper/220:28:452
204799258,15cyclictest7781-21perf20:35:010
204799251,14cyclictest28426-21sed21:20:160
204799250,12cyclictest31349-21basename20:15:020
205799241,13cyclictest0-21swapper/223:17:392
205799241,10cyclictest0-21swapper/223:00:032
205799240,12cyclictest0-21swapper/219:26:422
206399239,3cyclictest24490-21/usr/sbin/munin21:10:213
206399231,6cyclictest0-21swapper/322:57:233
205799231,9cyclictest0-21swapper/220:58:492
205799231,11cyclictest0-21swapper/223:52:152
205799231,10cyclictest0-21swapper/220:33:012
204799234,8cyclictest17175-21idleruntime00:35:180
206399227,5cyclictest0-21swapper/322:07:203
206399224,11cyclictest17291-21timerwakeupswit23:20:243
206399222,11cyclictest11146-21df_inode21:55:143
206399221,8cyclictest0-21swapper/321:44:023
205799221,9cyclictest0-21swapper/219:44:412
204799221,9cyclictest0-21swapper/023:20:130
2047992212,5cyclictest18723-21switchtime22:10:240
204799221,12cyclictest0-21swapper/022:05:130
204799220,5cyclictest7-21ksoftirqd/000:00:420
204799220,13cyclictest31144-21/usr/sbin/munin21:25:220
225022211,2sleep30-21swapper/319:54:463
206399214,3cyclictest0-21swapper/300:13:033
206399212,11cyclictest9880-21/usr/sbin/munin00:20:233
206399211,6cyclictest0-21swapper/323:48:233
206399211,6cyclictest0-21swapper/323:48:233
206399211,12cyclictest29014-21/usr/sbin/munin23:50:153
206399211,11cyclictest20743-21grep22:15:233
206399211,11cyclictest13447-21ntp_states20:45:213
204799215,13cyclictest4148-21tail22:50:260
204799214,5cyclictest28399-21/usr/sbin/munin22:35:220
204799213,10cyclictest21184-21/usr/sbin/munin21:05:180
204799210,5cyclictest87550irq/128-enp1s0-21:45:230
206399209,3cyclictest9240-21/usr/sbin/munin23:05:233
206399206,4cyclictest0-21swapper/323:59:173
206399205,5cyclictest0-21swapper/323:12:323
206399204,6cyclictest0-21swapper/300:09:473
206399204,3cyclictest0-21swapper/322:31:263
206399202,2cyclictest0-21swapper/321:27:203
206399201,8cyclictest4485-21cat19:15:023
206399201,6cyclictest0-21swapper/323:16:383
206399201,10cyclictest1654-21grep00:00:173
2063992010,4cyclictest0-21swapper/300:25:113
206399200,10cyclictest0-21swapper/323:27:283
204799209,3cyclictest13633-21/usr/sbin/munin23:15:140
204799206,3cyclictest7-21ksoftirqd/023:40:170
204799206,3cyclictest4214-21/usr/sbin/munin21:40:150
204799204,7cyclictest31935-21threads22:40:240
204799202,7cyclictest26652-21ntp_states21:15:210
204799201,6cyclictest8564-1kworker/u9:121:51:270
204799201,5cyclictest0-21swapper/021:13:410
2047992012,2cyclictest87550irq/128-enp1s0-22:15:160
204799201,11cyclictest7682-21/usr/sbin/munin00:15:180
204799201,11cyclictest31561-21/usr/sbin/munin20:15:220
204799201,11cyclictest26823-21/usr/sbin/munin23:45:240
204799201,11cyclictest26823-21/usr/sbin/munin23:45:240
204799200,12cyclictest22751-21cat23:35:130
249842191,2sleep30-21swapper/323:40:153
206399197,3cyclictest28156-21grep20:05:203
206399194,5cyclictest0-21swapper/321:50:413
206399191,7cyclictest18920-21grep22:10:253
2063991913,3cyclictest22879-21ntp_states22:20:223
206399191,12cyclictest0-21swapper/320:00:233
206399191,10cyclictest23159-21diskmemload22:45:033
206399191,10cyclictest11771-21/usr/sbin/munin19:30:213
206399191,10cyclictest0-21swapper/321:15:313
206399191,10cyclictest0-21swapper/300:17:503
206399190,8cyclictest0-21swapper/322:03:593
204799198,4cyclictest87550irq/128-enp1s0-22:45:270
204799198,3cyclictest808-21snmpd19:29:210
204799197,3cyclictest14276-21/usr/sbin/munin00:30:130
204799194,3cyclictest88950irq/132-enp2s0-22:02:050
204799192,9cyclictest22345-21sshd19:52:440
204799191,9cyclictest0-21swapper/022:20:190
204799191,5cyclictest0-21swapper/021:58:270
204799191,10cyclictest3602-21/usr/sbin/munin20:25:260
204799191,10cyclictest12074-21/usr/sbin/munin00:25:160
204799190,5cyclictest8564-1kworker/u9:123:50:130
206399187,5cyclictest24353-21diskstats22:25:133
206399185,3cyclictest579-21systemd-journal21:40:013
206399181,9cyclictest7611-21perl23:00:153
206399181,7cyclictest4592-21strings20:25:203
2063991813,3cyclictest22526-21tr23:35:123
2063991812,4cyclictest13971-21/usr/sbin/munin19:35:173
206399181,15cyclictest0-21swapper/321:05:153
206399181,11cyclictest87350irq/126-enp1s0-20:35:243
206399181,10cyclictest6413-21/usr/sbin/munin21:45:153
206399181,10cyclictest23159-21diskmemload21:32:283
204799189,2cyclictest9891-21acpi00:20:120
204799187,3cyclictest579-21systemd-journal19:40:200
204799183,2cyclictest87550irq/128-enp1s0-21:35:260
204799181,7cyclictest0-21swapper/000:14:140
2047991813,3cyclictest19433-21ntp_states19:45:220
204799181,15cyclictest0-21swapper/023:30:130
204799181,14cyclictest808-21snmpd19:30:180
204799181,14cyclictest18027-21/usr/sbin/munin23:25:130
109732181,2sleep00-21swapper/023:07:090
96222171,2sleep00-21swapper/020:35:480
302352171,13sleep30-21swapper/320:10:183
206399179,6cyclictest9603-21cpu19:25:123
206399174,3cyclictest0-21swapper/321:23:493
206399173,4cyclictest30594-21/usr/sbin/munin22:40:183
206399172,5cyclictest8564-1kworker/u9:122:53:113
206399171,9cyclictest839-21lldpd19:40:323
206399171,7cyclictest18633-21taskset20:58:393
206399171,6cyclictest4723-21ntp_states19:15:023
2063991712,3cyclictest32169-21gltestperf20:15:173
2063991712,3cyclictest20482-21grep21:00:263
206399171,15cyclictest0-21swapper/320:33:273
206399171,15cyclictest0-21swapper/319:48:483
206399171,10cyclictest0-21swapper/322:35:363
206399171,10cyclictest0-21swapper/320:51:373
204799174,6cyclictest81ktimersoftd/023:00:230
204799171,9cyclictest808-21snmpd20:23:450
204799171,6cyclictest7539-21cstates19:20:120
204799171,14cyclictest0-21swapper/019:15:060
2047991710,4cyclictest10715-21grep20:40:150
120052171,2sleep30-21swapper/320:43:133
206399162,5cyclictest23823-21ntp_states19:55:213
2063991611,3cyclictest1375-21/usr/sbin/munin20:20:193
2063991610,4cyclictest0-21swapper/319:20:173
204799166,3cyclictest0-21swapper/019:10:110
204799163,2cyclictest12450-21cpuspeed_turbos20:45:120
204799161,6cyclictest0-21swapper/022:55:540
204799161,4cyclictest11-21rcuc/019:55:180
2047991612,3cyclictest0-21swapper/020:07:100
2047991612,3cyclictest0-21swapper/020:07:100
204799151,5cyclictest26362-21timerwakeupswit20:00:230
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional