You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-08 - 20:24

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3s.osadl.org (updated Thu Jan 08, 2026 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
260722243166,25sleep20-21swapper/207:06:532
262022215165,38sleep30-21swapper/307:08:353
243582202164,25sleep10-21swapper/107:05:111
241472200166,22sleep00-21swapper/007:05:020
117252570,2sleep10-21swapper/107:45:251
117252570,2sleep10-21swapper/107:45:251
167792560,5sleep02639399cyclictest10:32:580
41292540,4sleep011-21rcuc/011:20:220
33182490,4sleep121-21rcuc/110:00:211
56872330,3sleep341-21ksoftirqd/308:50:163
26401993015,10cyclictest0-21swapper/110:10:511
26401992917,8cyclictest0-21swapper/112:26:231
2640799283,6cyclictest884-21runrttasks12:26:232
26401992816,3cyclictest27905-21cat09:40:191
26393992814,9cyclictest0-21swapper/009:30:120
108302280,4sleep02639399cyclictest11:35:230
2640799275,7cyclictest14777-21kworker/u8:212:14:022
2640799271,4cyclictest0-21swapper/209:40:192
26401992715,7cyclictest0-21swapper/110:35:471
26401992715,6cyclictest885-21sssd_nss10:35:011
26401992714,8cyclictest0-21swapper/110:21:241
2640199271,20cyclictest0-21swapper/110:42:371
2640199270,16cyclictest30220-21kernelversion12:20:161
2640799261,6cyclictest0-21swapper/211:43:502
2640799261,5cyclictest0-21swapper/210:50:052
2640799261,5cyclictest0-21swapper/210:06:512
2640799261,5cyclictest0-21swapper/209:27:512
2640799261,11cyclictest0-21swapper/212:04:442
26401992616,6cyclictest0-21swapper/111:20:131
26401992615,7cyclictest0-21swapper/112:05:151
26401992614,8cyclictest0-21swapper/111:27:461
2639399261,5cyclictest0-21swapper/010:29:460
2641399253,16cyclictest13525-21diskmemload11:52:003
2641399250,3cyclictest0-21swapper/310:15:393
2641399250,3cyclictest0-21swapper/310:15:393
2640799251,7cyclictest0-21swapper/211:21:032
2640799251,7cyclictest0-21swapper/209:52:292
2640799251,6cyclictest0-21swapper/210:10:512
26407992514,8cyclictest16780-21sshd10:32:582
2640799251,11cyclictest0-21swapper/212:24:142
26401992515,3cyclictest0-21swapper/110:06:591
26401992512,8cyclictest0-21swapper/112:01:461
26401992512,8cyclictest0-21swapper/110:27:041
2640199251,22cyclictest0-21swapper/108:28:041
2641399248,13cyclictest87150irq/126-enp1s0-12:23:583
2641399246,5cyclictest0-21swapper/310:24:553
2641399243,5cyclictest0-21swapper/309:15:223
2641399241,4cyclictest0-21swapper/311:57:403
2641399240,6cyclictest0-21swapper/311:40:253
2641399240,5cyclictest0-21swapper/311:15:463
2640799241,7cyclictest0-21swapper/210:35:472
26407992417,4cyclictest7492-21grep07:35:242
2640799241,6cyclictest0-21swapper/212:05:152
2640799241,5cyclictest0-21swapper/212:38:242
26401992415,3cyclictest0-21swapper/109:52:291
26401992413,7cyclictest0-21swapper/111:31:031
26401992413,3cyclictest9010-21gltestperf10:15:121
26401992413,3cyclictest9010-21gltestperf10:15:121
26401992412,8cyclictest0-21swapper/112:37:591
26401992411,9cyclictest0-21swapper/110:45:081
26401992410,9cyclictest0-21swapper/112:31:441
2640199240,9cyclictest0-21swapper/108:45:011
2639399244,3cyclictest87150irq/126-enp1s0-11:25:150
2641399233,6cyclictest0-21swapper/311:32:503
2641399233,11cyclictest0-21swapper/310:09:433
2641399231,9cyclictest0-21swapper/312:35:413
2641399231,6cyclictest0-21swapper/310:35:543
2641399231,4cyclictest9-21rcu_preempt11:01:393
2641399230,8cyclictest0-21swapper/312:00:053
2640799233,7cyclictest17930-40dnf11:52:002
2640799231,7cyclictest0-21swapper/211:31:032
2640799231,7cyclictest0-21swapper/211:02:062
2640799231,3cyclictest22538-21/usr/sbin/munin09:30:122
2640799231,3cyclictest0-21swapper/212:16:012
2640199239,3cyclictest29674-21cstates07:15:131
2640199237,5cyclictest16626-21hddtemp_smartct09:15:151
26401992314,3cyclictest0-21swapper/110:50:051
26401992314,3cyclictest0-21swapper/109:27:511
26401992313,3cyclictest0-21swapper/111:01:231
26401992313,3cyclictest0-21swapper/109:20:151
2640199231,19cyclictest0-21swapper/111:10:101
2640199231,10cyclictest0-21swapper/111:45:011
26401992310,9cyclictest0-21swapper/112:16:071
2639399239,2cyclictest0-21swapper/009:50:180
2639399233,11cyclictest18877-21seq08:04:580
2639399231,2cyclictest0-21swapper/011:55:260
2639399231,10cyclictest3772-21ls10:05:010
2641399226,4cyclictest23484-21smartctl12:05:143
2641399224,8cyclictest0-21swapper/311:35:073
2641399223,6cyclictest0-21swapper/309:32:023
2641399221,12cyclictest0-21swapper/310:25:123
2641399220,8cyclictest0-21swapper/309:23:073
2641399220,6cyclictest0-21swapper/309:26:353
2640799223,3cyclictest30-21rcuc/210:45:082
2640799221,6cyclictest32-21ksoftirqd/211:46:332
2640799221,6cyclictest0-21swapper/209:57:572
2640799221,6cyclictest0-21swapper/209:20:152
2640799221,5cyclictest0-21swapper/212:30:082
2640799221,3cyclictest9009-21gltestperf10:15:122
2640799221,3cyclictest9009-21gltestperf10:15:122
2640799221,3cyclictest0-21swapper/211:19:422
2640799221,3cyclictest0-21swapper/210:27:172
2640799221,3cyclictest0-21swapper/209:11:442
2640799221,12cyclictest16912-21/usr/sbin/munin08:00:182
2640199229,9cyclictest0-21swapper/111:58:451
2640199228,11cyclictest25895-21latency12:10:171
2640199221,6cyclictest0-21swapper/109:46:331
2640199221,3cyclictest10012-21taskset07:44:051
26401992213,3cyclictest0-21swapper/109:30:381
26401992212,4cyclictest0-21swapper/111:46:331
2640199221,19cyclictest0-21swapper/111:50:201
2640199221,19cyclictest0-21swapper/111:06:501
26401992210,8cyclictest0-21swapper/111:16:571
26401992210,8cyclictest0-21swapper/109:55:201
2639399228,2cyclictest0-21swapper/011:33:580
2639399225,7cyclictest87150irq/126-enp1s0-11:50:230
2639399225,5cyclictest13788-21munin-run11:45:010
2639399223,10cyclictest21405-21ntp_offset09:25:200
2639399223,10cyclictest15972-21crond11:50:010
2639399222,8cyclictest15239-21unixbench-2d09:10:250
2639399222,2cyclictest87150irq/126-enp1s0-08:27:350
26393992212,3cyclictest11317-21sshd09:04:580
26393992212,3cyclictest11317-21sshd09:04:580
2639399221,19cyclictest0-21swapper/007:19:580
2639399221,13cyclictest0-21swapper/010:50:180
2639399221,10cyclictest31634-21memory08:35:170
2639399221,10cyclictest0-21swapper/011:07:590
2639399220,9cyclictest0-21swapper/012:30:190
2639399220,3cyclictest11-21rcuc/012:36:100
320682210,10sleep339-21rcuc/307:20:143
2641399213,6cyclictest0-21swapper/309:47:553
2641399211,7cyclictest0-21swapper/310:30:593
2641399211,6cyclictest87150irq/126-enp1s0-12:33:213
2641399211,6cyclictest0-21swapper/312:11:493
2641399211,5cyclictest25415-21smartctl09:35:153
2641399211,12cyclictest0-21swapper/310:00:433
2641399211,11cyclictest0-21swapper/310:13:313
2641399210,10cyclictest0-21swapper/309:50:123
2640799217,4cyclictest19403-21netstat11:55:182
2640799214,3cyclictest30-21rcuc/211:25:152
2640799212,9cyclictest27972-21timerandwakeup07:10:242
2640799212,6cyclictest4735-21munin-run08:50:012
2640799211,6cyclictest0-21swapper/211:36:322
2640799211,4cyclictest14713-21/usr/sbin/munin07:55:192
2640799211,4cyclictest0-21swapper/211:11:232
2640799211,4cyclictest0-21swapper/210:01:102
2640799211,3cyclictest0-21swapper/209:17:192
2640799211,2cyclictest0-21swapper/209:49:272
26407992110,3cyclictest87150irq/126-enp1s0-10:40:132
2640199216,4cyclictest15657-21/usr/sbin/munin07:55:181
2640199216,3cyclictest580-21systemd-journal08:25:011
2640199211,6cyclictest0-21swapper/109:10:281
2640199211,3cyclictest0-21swapper/109:02:571
2640199211,3cyclictest0-21swapper/109:02:571
2640199211,18cyclictest0-21swapper/109:38:151
2640199211,18cyclictest0-21swapper/107:28:481
2639399214,7cyclictest0-21swapper/012:18:030
2639399213,5cyclictest87350irq/128-enp1s0-09:45:020
2639399211,8cyclictest25745-21NetworkManager08:20:420
2639399211,3cyclictest746-21/usr/sbin/munin11:15:150
2639399211,13cyclictest0-21swapper/012:20:350
2639399211,11cyclictest0-21swapper/012:03:260
2639399211,11cyclictest0-21swapper/010:22:240
2639399211,11cyclictest0-21swapper/010:10:150
2639399211,10cyclictest19419-21seq08:09:030
2641399208,3cyclictest19726-21/usr/sbin/munin08:10:193
2641399205,5cyclictest4287-21grep08:45:243
2641399205,5cyclictest0-21swapper/311:49:463
2641399204,2cyclictest87150irq/126-enp1s0-11:20:143
2641399202,11cyclictest0-21swapper/309:40:093
2641399201,7cyclictest401ktimersoftd/307:35:003
26413992015,2cyclictest580-21systemd-journal09:09:583
2641399201,4cyclictest1103-21taskset12:29:003
2641399201,11cyclictest27205-21/usr/sbin/munin12:15:203
2641399201,11cyclictest0-21swapper/310:56:443
2641399201,11cyclictest0-21swapper/310:50:483
2641399201,10cyclictest12516-21/usr/sbin/munin07:50:193
2641399201,10cyclictest0-21swapper/309:10:103
2641399200,9cyclictest19361-21kworker/u8:210:43:393
2641399200,8cyclictest0-21swapper/311:10:173
2641399200,7cyclictest0-21swapper/311:28:273
2640799205,7cyclictest9532-21cstates09:00:122
2640799205,7cyclictest9532-21cstates09:00:122
2640799205,4cyclictest30500-21awk08:35:002
2640799202,6cyclictest6592-40dnf08:50:542
2640799201,3cyclictest0-21swapper/210:55:522
2640799201,3cyclictest0-21swapper/209:35:432
2640199209,7cyclictest0-21swapper/110:56:251
2640199201,9cyclictest20473-21if_enp2s008:10:131
26401992016,3cyclictest0-21swapper/108:06:471
2640199201,3cyclictest0-21swapper/107:36:171
26401992011,3cyclictest0-21swapper/111:37:371
2639399206,6cyclictest87150irq/126-enp1s0-08:40:540
2639399204,12cyclictest87150irq/126-enp1s0-11:10:530
2639399202,8cyclictest87350irq/128-enp1s0-07:10:180
2639399202,8cyclictest0-21swapper/010:35:110
2639399202,10cyclictest19210-21ntp_states09:20:200
2639399201,9cyclictest22240-21hddtemp_smartct10:45:130
2639399201,8cyclictest11-21rcuc/011:05:000
2639399201,7cyclictest0-21swapper/008:18:540
2639399201,6cyclictest0-21swapper/010:59:390
2639399201,6cyclictest0-21swapper/009:56:170
2639399201,3cyclictest0-21swapper/009:15:140
2639399201,16cyclictest3731-21/usr/sbin/munin07:30:120
2641399198,3cyclictest19361-21kworker/u8:211:05:243
2641399197,3cyclictest21929-21/usr/sbin/munin08:15:243
2641399191,5cyclictest23540-21munin-run10:50:013
2641399191,5cyclictest0-21swapper/308:30:183
26413991914,3cyclictest0-21swapper/307:15:033
26413991913,3cyclictest10415-21ntp_states09:00:213
26413991913,3cyclictest10415-21ntp_states09:00:213
2641399191,11cyclictest0-21swapper/309:56:483
2640799191,6cyclictest32674-21chrt08:39:552
2640799191,6cyclictest0-21swapper/210:20:272
2640799191,3cyclictest0-21swapper/211:09:412
2640199197,2cyclictest0-21swapper/108:50:141
2640199194,12cyclictest0-21swapper/108:15:151
26401991915,3cyclictest0-21swapper/109:05:131
26401991915,3cyclictest0-21swapper/108:48:201
2640199191,3cyclictest0-21swapper/107:50:091
2640199191,17cyclictest0-21swapper/108:59:551
2640199191,17cyclictest0-21swapper/108:37:431
2640199191,17cyclictest0-21swapper/107:34:001
2640199191,17cyclictest0-21swapper/107:22:201
2640199190,18cyclictest0-21swapper/108:30:261
2639399196,3cyclictest28524-21/usr/sbin/munin08:30:130
2639399196,3cyclictest11-21rcuc/009:45:160
2639399194,3cyclictest25783-21if_err_enp2s012:10:160
2639399193,6cyclictest7-21ksoftirqd/008:10:590
2639399192,8cyclictest31688-21/usr/sbin/munin07:20:250
2639399191,6cyclictest0-21swapper/009:35:160
2639399191,6cyclictest0-21swapper/008:56:570
2639399191,4cyclictest3124-21users07:25:260
2639399191,3cyclictest0-21swapper/010:15:180
2639399191,3cyclictest0-21swapper/010:15:180
2639399191,12cyclictest0-21swapper/010:41:350
2639399191,12cyclictest0-21swapper/008:50:110
2639399191,11cyclictest5926-21/usr/sbin/munin07:35:170
2639399191,10cyclictest0-21swapper/012:29:220
181582191,14sleep118200-21ls08:00:261
2641399187,3cyclictest0-21swapper/308:22:593
2641399185,4cyclictest7964-21awk08:55:183
2641399183,4cyclictest1185-21grep08:40:153
2641399181,7cyclictest28291-21latency_hist08:30:013
26413991814,2cyclictest39-21rcuc/307:15:023
26413991813,3cyclictest15948-21find07:55:223
2641399181,10cyclictest0-21swapper/307:47:523
2641399181,10cyclictest0-21swapper/307:47:523
2640799184,7cyclictest29487-21taskset07:15:082
2640799184,3cyclictest87150irq/126-enp1s0-07:20:122
2640799184,3cyclictest3731-21/usr/sbin/munin07:30:222
2640799184,3cyclictest22423-21grep08:15:152
2640799183,3cyclictest814-21snmpd08:28:002
2640799181,9cyclictest814-21snmpd08:07:562
2640799181,9cyclictest7156-21/usr/sbin/munin08:55:182
2640799181,4cyclictest11557-21/usr/sbin/munin09:05:232
2640799181,15cyclictest0-21swapper/207:46:002
2640799181,15cyclictest0-21swapper/207:46:002
2640199181,16cyclictest0-21swapper/107:12:131
2639399185,3cyclictest87350irq/128-enp1s0-12:05:170
2639399185,3cyclictest5481-21unixbench-2d10:05:250
2639399182,4cyclictest14945-21cstates07:55:110
2639399182,4cyclictest13776-40dnf07:50:230
2639399181,7cyclictest12502-21cat07:50:010
133622181,2sleep00-21swapper/009:08:030
2641399179,5cyclictest31761-21grep08:35:193
2641399171,7cyclictest5933-21/usr/sbin/munin07:35:103
2641399171,6cyclictest17978-21ntp_states08:00:233
26413991711,3cyclictest9596-21grep07:40:243
26413991710,4cyclictest19510-21turbostat08:10:013
2640799172,5cyclictest1216-21hddtemp_smartct08:40:152
2640799171,6cyclictest1907-21uniq07:25:152
2640799171,3cyclictest0-21swapper/207:50:162
2640799171,2cyclictest0-21swapper/208:20:152
2639399172,13cyclictest2759-21/usr/sbin/munin08:45:230
2639399172,12cyclictest8122-21/usr/sbin/munin07:40:110
2639399170,8cyclictest0-21swapper/007:45:110
197192170,2sleep20-21swapper/208:10:032
2641399169,4cyclictest1654-21cut07:25:123
26407991610,3cyclictest8122-21/usr/sbin/munin07:40:132
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional