You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-15 - 23:10

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Fri May 15, 2026 12:44:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
228742241166,24sleep30-21swapper/307:06:073
231332240164,24sleep20-21swapper/207:09:282
231172202165,24sleep10-21swapper/107:09:151
231212201164,25sleep00-21swapper/007:09:190
799621520,4sleep12326599cyclictest10:12:421
452121450,3sleep12326599cyclictest11:20:011
23269991171,114cyclictest0-21swapper/208:18:382
23269998986,2cyclictest0-21swapper/207:15:062
23269998985,3cyclictest0-21swapper/208:45:542
23269996354,8cyclictest0-21swapper/209:53:442
18922390,21sleep11887-21ntp_kernel_err12:25:181
170972360,3sleep10-21swapper/111:45:261
170972360,3sleep10-21swapper/111:45:261
23257993510,11cyclictest4764-21idleruntime-cro07:40:020
23275993421,3cyclictest0-21swapper/309:11:013
2325799344,14cyclictest30799-21munin-run12:20:010
2327599311,11cyclictest0-21swapper/309:52:423
2327599301,18cyclictest0-21swapper/311:25:053
2327599291,11cyclictest0-21swapper/312:29:343
23275992814,4cyclictest0-21swapper/312:39:223
2327599280,14cyclictest0-21swapper/307:35:143
2326599281,24cyclictest21375-21munin-run10:45:011
2327599271,12cyclictest0-21swapper/309:15:263
2326599273,15cyclictest87550irq/128-enp1s0-08:03:591
23265992715,9cyclictest13536-21latency_hist08:00:011
23265992713,8cyclictest3649-40dnf10:03:241
2326599271,20cyclictest0-21swapper/112:14:531
44472260,4sleep07-21ksoftirqd/012:30:230
2327599261,12cyclictest0-21swapper/310:48:103
2327599260,12cyclictest0-21swapper/310:50:063
23265992621,3cyclictest0-21swapper/109:30:131
2325799263,10cyclictest28948-21awk08:35:020
2327599259,10cyclictest0-21swapper/310:39:183
23275992518,4cyclictest5972-21/usr/sbin/munin11:20:233
2327599251,11cyclictest0-21swapper/310:10:223
2326999251,4cyclictest23326-21kworker/u8:011:47:522
2326999251,4cyclictest23326-21kworker/u8:011:47:522
2326599253,3cyclictest773-21NetworkManager07:22:011
2326599252,19cyclictest13200-21munin-run11:40:011
2326599250,23cyclictest0-21swapper/110:38:241
2325799258,14cyclictest958-21perf08:45:010
2327599249,3cyclictest16973-21latency_hist10:35:023
2327599249,3cyclictest16973-21latency_hist10:35:023
2327599247,3cyclictest25041-21cstates08:25:133
2327599241,9cyclictest0-21swapper/309:46:023
23275992412,7cyclictest0-21swapper/311:53:293
2327599241,11cyclictest0-21swapper/311:44:213
2327599241,10cyclictest0-21swapper/310:29:423
23265992418,3cyclictest26671-21hddtemp_smartct10:55:161
2326599241,16cyclictest0-21swapper/109:10:401
2326599241,11cyclictest21985-21sh12:00:021
2325799241,11cyclictest9787-21perf09:05:000
2325799241,11cyclictest6620-21perf11:25:010
2325799241,11cyclictest28330-21munin-run07:20:010
2327599236,3cyclictest11573-21diskstats11:35:133
2327599236,1cyclictest0-21swapper/310:40:263
2327599232,18cyclictest0-21swapper/310:00:183
2327599231,13cyclictest495-21jbd2/dm-0-809:35:183
2327599231,11cyclictest0-21swapper/311:00:533
2327599230,9cyclictest0-21swapper/307:10:133
2326999235,6cyclictest0-21swapper/209:10:232
2326999231,20cyclictest0-21swapper/209:01:122
2326999231,11cyclictest20626-21gltestperf11:55:162
2326599238,4cyclictest0-21swapper/109:50:121
2326599236,9cyclictest0-21swapper/112:17:321
2326599231,4cyclictest580-21ntp_states09:55:201
2326599231,4cyclictest15460-21ntp_states09:15:211
2326599231,19cyclictest4533-21pmu-power08:50:221
2326599231,16cyclictest0-21swapper/108:47:081
2326599231,10cyclictest25770-21sh10:55:011
276082220,2sleep10-21swapper/107:15:261
2327599228,3cyclictest0-21swapper/307:27:383
2327599225,3cyclictest10534-21latency_hist10:20:023
2327599221,5cyclictest0-21swapper/309:32:353
2327599221,11cyclictest0-21swapper/311:12:013
2327599221,11cyclictest0-21swapper/309:25:063
23275992210,3cyclictest0-21swapper/309:20:183
2327599220,16cyclictest0-21swapper/311:17:413
2326599228,9cyclictest0-21swapper/108:56:371
2326599227,9cyclictest0-21swapper/112:30:081
2326599226,8cyclictest27579-21/usr/sbin/munin09:45:211
2326599226,2cyclictest0-21swapper/109:02:551
2326599225,8cyclictest23946-21if_enp2s009:35:161
2326599225,10cyclictest1295-21ntpq11:10:211
2326599222,15cyclictest87650irq/129-enp1s0-11:34:051
23265992218,2cyclictest12579-21crond10:25:001
2326599221,5cyclictest23326-21kworker/u8:011:21:171
2326599221,3cyclictest24542-21munin-run08:25:021
23265992213,6cyclictest7477-21gltestperf11:25:171
2326599221,14cyclictest0-21swapper/111:42:341
2326599221,13cyclictest0-21swapper/112:05:401
2326599221,13cyclictest0-21swapper/110:30:071
2326599221,13cyclictest0-21swapper/110:30:071
2326599221,12cyclictest28998-21ls08:35:021
2326599220,20cyclictest0-21swapper/110:08:311
2325799228,3cyclictest24136-21kworker/0:212:06:130
166782220,1sleep10-21swapper/109:20:131
119342221,18sleep212168-21head07:55:152
2327599218,3cyclictest0-21swapper/308:32:213
2327599217,3cyclictest0-21swapper/310:21:223
2327599210,13cyclictest0-21swapper/312:03:213
2326999215,3cyclictest12084-21cat09:10:022
2326999211,7cyclictest32424-21ntp_states08:40:222
2326999211,7cyclictest13424-21/usr/sbin/munin11:40:282
2326999211,5cyclictest4411-21munin-run11:20:002
2326999211,11cyclictest0-21swapper/208:28:362
2326599217,8cyclictest0-21swapper/108:38:091
2326599211,9cyclictest23580-21munin-run10:50:011
2326599211,7cyclictest0-21swapper/111:05:241
23265992117,3cyclictest0-21swapper/108:16:511
23265992117,2cyclictest19252-21timerwakeupswit11:50:251
2326599211,2cyclictest26645-21smart_sda09:40:221
2326599211,2cyclictest0-21swapper/112:35:211
2326599211,18cyclictest0-21swapper/110:26:511
2326599211,18cyclictest0-21swapper/107:38:241
2326599211,15cyclictest0-21swapper/108:41:201
2326599211,14cyclictest0-21swapper/112:01:331
2326599211,10cyclictest32225-21unixbench-2d07:25:251
2326599210,5cyclictest0-21swapper/109:26:041
2325799218,7cyclictest87350irq/126-enp1s0-12:21:360
2325799216,6cyclictest0-21swapper/009:20:430
2325799213,16cyclictest32390-21idleruntime-cro11:10:020
2325799212,9cyclictest23167-21kworker/0:009:58:010
2325799212,11cyclictest12192-21/usr/sbin/munin09:10:210
2325799211,9cyclictest23167-21kworker/0:009:38:230
2325799211,12cyclictest6558-21df_inode10:10:140
2325799211,11cyclictest30304-21cat11:05:020
2327599205,9cyclictest0-21swapper/307:40:153
2327599201,5cyclictest0-21swapper/311:58:143
2327599201,13cyclictest0-21swapper/308:55:243
2327599201,11cyclictest0-21swapper/312:10:063
2327599201,10cyclictest0-21swapper/308:16:213
2326999207,3cyclictest30-21rcuc/212:30:242
2326999206,2cyclictest28912-21basename11:00:162
2326999205,4cyclictest11963-21diskmemload11:10:112
2326999205,3cyclictest8188-21cat11:25:252
2326999205,3cyclictest19600-21kworker/2:209:25:142
2326999204,5cyclictest1227-21fschecks_count12:25:142
2326999201,8cyclictest0-21swapper/209:55:262
2326999201,7cyclictest5419-21cstates12:35:112
2326999201,7cyclictest25138-21idleruntime12:05:172
2326999201,7cyclictest0-21swapper/209:30:282
2326999201,7cyclictest0-21swapper/208:20:022
2326999201,6cyclictest2158-21diskstats10:00:152
2326999201,6cyclictest17821-21/usr/sbin/munin11:50:192
23269992014,3cyclictest32081-21ntpq12:20:222
2326999200,19cyclictest0-21swapper/212:19:182
2326599203,5cyclictest21-21rcuc/107:45:161
2326599201,9cyclictest0-21swapper/110:15:211
23265992016,3cyclictest0-21swapper/109:06:081
23265992016,3cyclictest0-21swapper/108:07:341
2326599201,3cyclictest0-21swapper/111:00:221
2326599201,17cyclictest26121-21timerandwakeup08:25:261
2326599201,17cyclictest0-21swapper/107:12:001
2326599201,11cyclictest485-21/usr/sbin/munin07:30:221
2326599200,18cyclictest0-21swapper/108:10:021
2325799208,4cyclictest0-21swapper/010:35:230
2325799207,4cyclictest24255-21cat07:10:170
2325799206,4cyclictest0-21swapper/011:25:240
2325799205,6cyclictest0-21swapper/009:54:200
2325799202,9cyclictest6204-21kworker/0:010:17:580
2325799202,8cyclictest20725-21kworker/0:109:30:130
2325799201,6cyclictest0-21swapper/011:56:370
2325799201,16cyclictest3399-21users10:00:260
2325799201,11cyclictest20369-21/usr/sbin/munin08:15:240
2325799201,11cyclictest10608-21/usr/sbin/munin10:20:220
2325799200,9cyclictest15723-21kworker/0:008:05:520
2325799200,6cyclictest9066-1kworker/u9:009:41:040
58952191,3sleep30-21swapper/310:09:013
2327599191,6cyclictest0-21swapper/311:05:223
2327599191,2cyclictest87550irq/128-enp1s0-08:12:133
2327599191,1cyclictest0-21swapper/312:22:103
2327599191,11cyclictest0-21swapper/311:31:333
2327599191,11cyclictest0-21swapper/310:57:003
2327599191,10cyclictest27022-21sshd09:40:543
2327599191,10cyclictest11571-21/usr/sbin/munin07:55:243
2326999197,3cyclictest87350irq/126-enp1s0-10:25:162
2326999196,3cyclictest87350irq/126-enp1s0-12:10:152
2326999195,3cyclictest9873-21grep11:30:172
2326999194,4cyclictest15271-21cat10:30:122
2326999194,4cyclictest15271-21cat10:30:122
2326999193,9cyclictest0-21swapper/210:36:502
2326999191,7cyclictest15757-21perf08:05:002
2326999191,6cyclictest19399-21/usr/sbin/munin10:40:182
2326999191,6cyclictest0-21swapper/211:08:032
2326999191,5cyclictest29159-21/usr/sbin/munin08:35:172
23265991916,2cyclictest0-21swapper/107:40:441
23265991915,3cyclictest0-21swapper/107:51:031
2326599191,10cyclictest31110-21ls12:20:131
2325799198,2cyclictest27998-21kworker/0:112:14:290
2325799197,2cyclictest4853-21kworker/0:012:36:570
2325799193,7cyclictest0-21swapper/010:25:250
2325799192,9cyclictest808-21snmpd10:08:390
2325799191,12cyclictest0-21swapper/008:00:230
2325799191,11cyclictest0-21swapper/010:55:080
2325799191,10cyclictest0-21swapper/011:15:530
2325799190,6cyclictest0-21swapper/011:10:300
17982190,3sleep30-21swapper/307:30:243
173952191,2sleep00-21swapper/011:47:370
173952191,2sleep00-21swapper/011:47:370
92632180,4sleep230-21rcuc/207:50:002
91072181,2sleep00-21swapper/007:49:400
50732180,5sleep22326999cyclictest11:20:132
2327599187,3cyclictest9066-1kworker/u9:008:02:523
2327599187,3cyclictest15976-21/usr/sbin/munin08:05:253
2327599184,2cyclictest87450irq/127-enp1s0-08:35:213
2327599183,6cyclictest1172-21/usr/sbin/munin08:45:183
2327599183,3cyclictest10079-21cat09:05:133
2327599182,7cyclictest87450irq/127-enp1s0-12:18:343
2327599181,4cyclictest810-21sshd07:17:543
2327599181,3cyclictest3969-21memory12:30:183
2326999184,3cyclictest87350irq/126-enp1s0-10:15:222
2326999184,3cyclictest87350irq/126-enp1s0-09:15:172
2326999184,3cyclictest27021-21sshd09:40:542
2326999183,4cyclictest11963-21diskmemload10:12:472
2326999183,3cyclictest29722-21date09:50:022
2326999183,3cyclictest11163-21perl10:20:142
2326999182,6cyclictest0-21swapper/209:35:192
2326999182,4cyclictest18167-21/usr/sbin/munin08:10:102
2326999182,14cyclictest0-21swapper/208:54:232
2326999181,6cyclictest11224-21/usr/sbin/munin11:35:192
2326999181,5cyclictest16591-21/usr/sbin/munin09:20:242
2326999181,4cyclictest23462-21/usr/sbin/munin07:10:172
2326999181,4cyclictest0-21swapper/212:00:182
2326999181,4cyclictest0-21swapper/210:06:282
23269991813,3cyclictest25284-21grep10:50:242
23269991812,3cyclictest29104-21tune2fs07:20:162
2326999181,14cyclictest26660-21cut10:55:152
2326999181,10cyclictest0-21swapper/207:34:472
2325799187,3cyclictest0-21swapper/011:52:180
2325799184,5cyclictest0-21swapper/012:25:050
2325799184,3cyclictest14744-21chrt09:15:140
2325799183,3cyclictest23326-21kworker/u8:010:47:070
2325799181,9cyclictest808-21snmpd08:48:100
2325799181,8cyclictest23248-21cyclictest09:46:260
23257991812,4cyclictest0-21swapper/009:10:000
23257991812,3cyclictest10408-21ntpq07:50:200
2325799181,11cyclictest0-21swapper/011:33:280
2325799180,6cyclictest0-21swapper/009:25:090
2327599171,14cyclictest4042-21cut08:50:153
2327599171,13cyclictest32626-21cut09:55:163
2327599171,13cyclictest15779-21grep11:45:123
2327599171,13cyclictest15779-21grep11:45:123
23275991710,4cyclictest0-21swapper/312:05:143
2326999174,3cyclictest19880-21kworker/u8:007:25:272
2326999172,5cyclictest0-21swapper/208:10:002
2326999171,5cyclictest579-21systemd-journal07:53:572
2326999171,5cyclictest27044-21cpuspeed_turbos08:30:122
2326999171,5cyclictest22857-21smart_sda10:45:242
2326999171,4cyclictest7142-21head08:55:252
2326999171,4cyclictest5785-21irqstats07:40:162
2325799174,7cyclictest0-21swapper/008:59:560
2325799173,3cyclictest0-21swapper/011:44:130
2325799172,6cyclictest29180-21egrep08:35:110
2325799172,3cyclictest0-21swapper/010:40:040
23257991712,3cyclictest22900-21df08:20:140
2325799171,13cyclictest11224-21/usr/sbin/munin11:35:130
2325799171,13cyclictest0-21swapper/012:00:060
2325799171,10cyclictest0-21swapper/010:52:520
200372170,2sleep00-21swapper/008:13:520
302012160,3sleep00-21swapper/007:21:020
2327599167,3cyclictest0-21swapper/307:45:143
2327599162,3cyclictest0-21swapper/308:20:173
2327599162,11cyclictest7785-21/usr/sbin/munin09:00:253
2327599162,11cyclictest10025-21gltestperf07:50:153
23275991611,3cyclictest29769-21cut07:20:233
23269991611,3cyclictest2780-21/usr/sbin/munin07:35:132
2325799165,3cyclictest87350irq/126-enp1s0-07:45:010
2325799161,7cyclictest0-21swapper/007:30:230
2325799161,14cyclictest0-21swapper/007:58:260
2325799161,12cyclictest15534-21fschecks_time10:30:150
2325799161,12cyclictest15534-21fschecks_time10:30:150
23257991610,3cyclictest26195-21timerwakeupswit08:25:260
23275991510,3cyclictest31723-21sort08:40:143
2325799151,9cyclictest0-21swapper/008:52:530
23257991510,3cyclictest30806-21cpu07:25:130
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional