You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-26 - 04:57

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackcslot7s.osadl.org (updated Wed Aug 26, 2026 00:44:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
260738726546,13sleep20-21swapper/219:05:492
260741326444,14sleep30-21swapper/319:06:113
260736026442,15sleep00-21swapper/019:05:310
260758126041,14sleep10-21swapper/119:08:221
26234102500,3sleep20-21swapper/219:50:162
26426532440,4sleep10-21swapper/120:45:161
26431362410,3sleep3411rcuc/320:45:233
26244282380,3sleep12624427-21systemd-run19:55:001
260785999370,21cyclictest0-21swapper/220:55:172
27011432350,1sleep32701148-21needreboot23:30:203
139099320,4rtkit-daemon1389-21rtkit-daemon23:58:391
139099300,4rtkit-daemon1389-21rtkit-daemon20:32:201
26182462290,13sleep234-21ksoftirqd/219:35:212
260785499292,5cyclictest883-21dbus-daemon22:20:001
2607854992915,12cyclictest2640462-21sed20:40:011
2607854992910,17cyclictest2696245-21cat23:15:221
139099290,4rtkit-daemon1389-21rtkit-daemon23:56:210
2607859992811,6cyclictest2613254-21sed19:20:212
260785499288,18cyclictest2622541-21sed19:50:011
260785499282,24cyclictest2718765-21grep00:20:181
260785499282,24cyclictest2718765-21grep00:20:181
2607854992812,14cyclictest2661032-21head21:35:171
2607854992812,14cyclictest2609581-21sed19:15:001
260785499279,16cyclictest2677631-21cut22:25:011
260785499278,17cyclictest2701011-21sort23:30:181
260785499278,17cyclictest2627215-21processes20:00:201
2607854992715,10cyclictest2668756-21apt22:00:011
260785999268,16cyclictest2639465-21needreboot20:35:172
2607859992612,12cyclictest2653358-21latency_hist21:15:002
2607859992611,13cyclictest2710879-21dump-pmu-power00:00:012
260785499267,17cyclictest2712615-21cron00:09:561
260785499261,4cyclictest0-21swapper/123:40:121
260785499261,4cyclictest0-21swapper/121:40:111
260785499260,4cyclictest2699546-21grep23:25:191
260785499260,25cyclictest2678582-21wc22:25:231
139099260,4rtkit-daemon1389-21rtkit-daemon23:18:560
139099260,4rtkit-daemon1389-21rtkit-daemon21:54:350
139099260,4rtkit-daemon1389-21rtkit-daemon21:51:221
260785999259,4cyclictest2671258-21sed22:05:182
2607859992511,12cyclictest2682873-21/usr/sbin/munin22:40:212
260785999250,24cyclictest0-21swapper/221:19:442
260785999250,1cyclictest0-21swapper/219:25:472
260785499258,15cyclictest2687172-21processes22:50:211
260785499258,15cyclictest2686089-21dump-pmu-power22:50:001
260785499257,17cyclictest2690097-21ls23:00:131
260785499257,16cyclictest2616343-21sed19:30:201
260785499257,15cyclictest2718049-21latency_hist00:20:001
260785499256,17cyclictest2630694-21cat20:10:181
260785499255,6cyclictest883-21dbus-daemon22:40:001
260785499255,6cyclictest2641924-21seq20:44:241
260785499250,23cyclictest2618426-21timerandwakeup19:35:231
139099250,3rtkit-daemon1389-21rtkit-daemon19:52:090
260785999249,5cyclictest2660087-21cron21:39:552
260785999249,3cyclictest2695928-21needreboot23:15:182
260785999242,7cyclictest2723555-21apt00:35:122
260785999240,8cyclictest2704533-21taskset23:40:182
260785999240,23cyclictest0-21swapper/220:02:102
260785499248,15cyclictest26-21ksoftirqd/122:10:001
260785499247,16cyclictest2721181-21sshd00:28:481
260785499247,15cyclictest2708467-21proc_pri23:50:211
260785499246,16cyclictest2614115-21ls19:25:131
260785499246,15cyclictest2716913-21latency00:15:191
260785499245,5cyclictest2706287-21expr23:45:131
260785499242,20cyclictest2628601-21cstates20:05:141
260785499241,9cyclictest2651704-21apt21:10:121
260785499241,5cyclictest2723286-21perf00:35:001
2607854992410,13cyclictest2672470-21/usr/sbin/munin22:10:221
139099240,4rtkit-daemon1389-21rtkit-daemon20:20:510
139099240,4rtkit-daemon1389-21rtkit-daemon20:20:500
260786499230,21cyclictest3056-21PK-Backend21:10:203
260785999238,5cyclictest2664112-21expr21:45:132
260785999237,4cyclictest2637945-21grep20:30:232
260785999232,9cyclictest2669086-21sort22:00:122
260785999231,9cyclictest2681562-21hddtemp_smartct22:35:152
260785999231,7cyclictest2685492-21python322:45:202
260785999231,6cyclictest2693674-21cat23:10:152
260785499237,14cyclictest2656114-21timerandwakeup21:20:231
260785499237,14cyclictest2656114-21timerandwakeup21:20:231
260785499236,5cyclictest2657885-21cat21:25:221
260785499236,15cyclictest2698030-21cat23:20:221
260785499236,15cyclictest2698030-21cat23:20:221
260785499236,15cyclictest2694330-21grep23:10:221
260785499234,5cyclictest2703083-21smartctl23:35:201
260785499231,9cyclictest2665311-21perf21:50:001
260785499231,6cyclictest2631858-21apt20:15:121
260785499231,5cyclictest2644456-21/usr/sbin/munin20:50:211
260785499231,4cyclictest2716104-21idleruntime-cro00:15:001
260785499231,2cyclictest2679736-21expr22:30:141
2607854992311,11cyclictest2647738-21unin-run21:00:011
260785499230,6cyclictest2619697-21kernelversion19:40:171
260785499230,5cyclictest436-20systemd-journal22:05:011
260785199230,21cyclictest0-21swapper/023:40:120
2607864992220,1cyclictest3056-21PK-Backend20:18:433
2607864992219,2cyclictest3056-21PK-Backend21:02:093
260786499220,20cyclictest3056-21PK-Backend19:51:123
260785999229,3cyclictest978-21snmpd22:11:112
260785999227,3cyclictest2632467-21missed_timers20:15:192
260785999226,4cyclictest2689215-21sshd22:55:472
2607859992220,1cyclictest0-21swapper/223:26:032
2607859992220,1cyclictest0-21swapper/220:24:342
2607859992220,1cyclictest0-21swapper/220:24:342
260785999221,9cyclictest2678433-21ntp_states22:25:202
260785999221,8cyclictest2616599-21python319:30:232
260785999221,7cyclictest2657661-21open_inodes21:25:192
2607859992217,4cyclictest0-21swapper/221:02:132
260785999221,6cyclictest2608787-21ntp_states19:10:192
260785999221,19cyclictest2625529-21python319:55:212
260785999220,8cyclictest2717814-21cron00:19:592
260785999220,8cyclictest2717814-21cron00:19:592
260785999220,21cyclictest0-21swapper/223:07:182
260785999220,21cyclictest0-21swapper/220:47:462
260785999220,10cyclictest884-21gmain21:52:482
260785499227,13cyclictest2609972-21interrupts19:15:031
260785499225,3cyclictest2653816-21entropy21:15:161
260785499225,3cyclictest2624967-21cat19:55:131
260785499225,2cyclictest2649927-21cstates21:05:221
260785499221,7cyclictest2711593-21/usr/sbin/munin00:00:161
260785499221,5cyclictest2692073-21latency23:05:141
260785499221,5cyclictest2647948-21/usr/sbin/munin21:00:111
260785499221,4cyclictest0-21swapper/120:25:121
260785499221,3cyclictest0-21swapper/120:20:121
260785499221,3cyclictest0-21swapper/120:20:111
260785499220,7cyclictest2613880-21sed19:25:001
260785499220,6cyclictest2688940-21ps22:55:211
260785499220,5cyclictest2658611-21/usr/sbin/munin21:30:121
2607851992220,1cyclictest0-21swapper/019:42:460
2607851992218,1cyclictest0-21swapper/019:19:000
2607851992217,3cyclictest2614394-21hddtemp_smartct19:25:160
260785199220,21cyclictest0-21swapper/020:58:290
2607864992119,1cyclictest3056-21PK-Backend23:45:343
2607864992119,1cyclictest3056-21PK-Backend22:28:073
2607864992119,1cyclictest3056-21PK-Backend19:40:303
2607864992119,1cyclictest3056-21PK-Backend00:10:413
2607864992119,1cyclictest3056-21PK-Backend00:00:213
2607864992119,1cyclictest0-21swapper/300:37:103
2607864992118,1cyclictest3056-21PK-Backend23:50:133
2607864992118,1cyclictest3056-21PK-Backend23:15:113
2607864992117,2cyclictest3056-21PK-Backend21:35:013
260786499210,20cyclictest3056-21PK-Backend20:56:473
260786499210,20cyclictest3056-21PK-Backend20:35:263
260786499210,20cyclictest3056-21PK-Backend20:07:203
260786499210,19cyclictest3056-21PK-Backend22:01:513
260786499210,19cyclictest3056-21PK-Backend21:27:243
260786499210,19cyclictest3056-21PK-Backend21:16:083
260786499210,19cyclictest3056-21PK-Backend20:50:543
260785999217,3cyclictest2690561-21ntpq23:00:182
2607859992119,1cyclictest0-21swapper/223:39:092
2607859992119,1cyclictest0-21swapper/223:30:302
2607859992119,1cyclictest0-21swapper/223:24:142
2607859992119,1cyclictest0-21swapper/223:24:142
2607859992119,1cyclictest0-21swapper/222:53:122
2607859992119,1cyclictest0-21swapper/221:42:352
2607859992119,1cyclictest0-21swapper/221:34:092
2607859992119,1cyclictest0-21swapper/221:22:272
2607859992119,1cyclictest0-21swapper/221:22:272
2607859992119,1cyclictest0-21swapper/221:09:492
2607859992119,1cyclictest0-21swapper/220:41:172
2607859992119,1cyclictest0-21swapper/220:06:502
2607859992119,1cyclictest0-21swapper/200:34:592
260785999211,8cyclictest2713756-21python300:05:232
260785999211,7cyclictest2674766-21/usr/sbin/munin22:15:182
260785999211,6cyclictest2721290-21apt-get00:30:002
2607859992110,9cyclictest2619171-21date19:40:002
260785999210,20cyclictest0-21swapper/222:23:522
260785999210,20cyclictest0-21swapper/200:20:592
260785999210,20cyclictest0-21swapper/200:20:592
260785999210,19cyclictest3056-21PK-Backend21:55:252
2607854992117,3cyclictest3056-21PK-Backend22:41:261
260785499210,19cyclictest0-21swapper/100:36:461
260785199211,9cyclictest26890682timerandwakeup22:55:220
2607851992119,1cyclictest0-21swapper/023:01:400
2607851992119,1cyclictest0-21swapper/022:53:070
2607851992119,1cyclictest0-21swapper/022:34:290
2607851992119,1cyclictest0-21swapper/022:26:590
2607851992119,1cyclictest0-21swapper/022:20:160
2607851992119,1cyclictest0-21swapper/021:59:070
2607851992119,1cyclictest0-21swapper/021:24:260
2607851992119,1cyclictest0-21swapper/021:24:250
2607851992119,1cyclictest0-21swapper/020:36:060
2607851992119,1cyclictest0-21swapper/020:06:120
2607851992119,1cyclictest0-21swapper/000:33:180
2607851992119,1cyclictest0-21swapper/000:10:140
2607851992114,5cyclictest2621389-21if_err_enp2s019:45:170
260785199210,20cyclictest3056-21PK-Backend19:21:510
260785199210,20cyclictest0-21swapper/023:09:370
260785199210,20cyclictest0-21swapper/019:58:550
27224002200,4sleep33056-21PK-Backend00:30:183
2607864992018,1cyclictest3056-21PK-Backend23:24:383
2607864992018,1cyclictest3056-21PK-Backend23:24:373
2607864992018,1cyclictest3056-21PK-Backend21:22:443
2607864992018,1cyclictest3056-21PK-Backend21:22:433
2607864992018,1cyclictest3056-21PK-Backend20:10:113
2607864992018,1cyclictest3056-21PK-Backend19:55:513
2607864992018,1cyclictest3056-21PK-Backend19:35:163
2607864992018,1cyclictest3056-21PK-Backend19:30:493
2607864992018,1cyclictest3056-21PK-Backend19:28:133
2607864992018,1cyclictest3056-21PK-Backend19:17:183
2607864992018,1cyclictest3056-21PK-Backend00:17:113
2607864992018,1cyclictest3056-21PK-Backend00:17:113
2607864992018,1cyclictest0-21swapper/322:46:333
2607864992018,1cyclictest0-21swapper/322:44:093
2607864992017,2cyclictest3056-21PK-Backend23:42:423
2607864992017,2cyclictest3056-21PK-Backend22:10:503
2607864992017,2cyclictest3056-21PK-Backend00:06:013
260786499200,1cyclictest3056-21PK-Backend00:28:053
260786499200,1cyclictest3056-21PK-Backend00:24:273
260786499200,1cyclictest3056-21PK-Backend00:24:273
260786499200,19cyclictest3056-21PK-Backend23:28:033
260786499200,19cyclictest3056-21PK-Backend22:31:053
260786499200,19cyclictest3056-21PK-Backend20:24:383
260786499200,19cyclictest3056-21PK-Backend20:24:383
260785999207,3cyclictest978-21snmpd20:25:172
260785999202,4cyclictest2680298-21python322:30:212
2607859992018,1cyclictest0-21swapper/223:45:362
2607859992018,1cyclictest0-21swapper/220:51:322
2607859992018,1cyclictest0-21swapper/200:03:422
260785999200,19cyclictest0-21swapper/223:51:182
260785999200,19cyclictest0-21swapper/219:17:022
2607851992018,1cyclictest0-21swapper/022:43:490
2607851992018,1cyclictest0-21swapper/022:13:000
2607851992018,1cyclictest0-21swapper/022:04:350
2607851992018,1cyclictest0-21swapper/021:48:590
2607851992018,1cyclictest0-21swapper/021:28:320
2607851992018,1cyclictest0-21swapper/021:09:190
2607851992018,1cyclictest0-21swapper/020:48:030
2607851992018,1cyclictest0-21swapper/020:33:230
2607851992018,1cyclictest0-21swapper/020:10:520
260785199200,19cyclictest3056-21PK-Backend22:45:230
260785199200,19cyclictest0-21swapper/021:41:020
260785199200,19cyclictest0-21swapper/021:13:110
260785199200,19cyclictest0-21swapper/021:00:060
260785199200,19cyclictest0-21swapper/020:50:590
260785199200,19cyclictest0-21swapper/020:43:430
260785199200,19cyclictest0-21swapper/020:16:330
260785199200,19cyclictest0-21swapper/019:35:430
260785199200,19cyclictest0-21swapper/000:02:290
139099200,16rtkit-daemon2607845-21cyclictest20:01:290
2607864991917,1cyclictest3056-21PK-Backend21:53:053
2607864991917,1cyclictest3056-21PK-Backend20:44:253
2607864991917,1cyclictest3056-21PK-Backend20:34:043
2607864991916,1cyclictest3056-21PK-Backend22:22:593
2607864991913,4cyclictest978-21snmpd19:10:453
260786499190,1cyclictest3056-21PK-Backend23:57:303
260786499190,1cyclictest3056-21PK-Backend23:08:573
260786499190,1cyclictest3056-21PK-Backend21:57:463
260786499190,1cyclictest3056-21PK-Backend21:34:133
260786499190,1cyclictest3056-21PK-Backend21:05:543
260786499190,1cyclictest3056-21PK-Backend20:28:423
260786499190,1cyclictest3056-21PK-Backend20:01:523
260786499190,1cyclictest3056-21PK-Backend19:45:443
260786499190,1cyclictest0-21swapper/323:10:463
260786499190,1cyclictest0-21swapper/322:36:253
2607859991917,1cyclictest0-21swapper/220:10:412
2607859991917,1cyclictest0-21swapper/200:12:272
2607851991917,1cyclictest0-21swapper/023:13:150
2607851991913,5cyclictest2654006-21load21:15:200
2607851991912,6cyclictest2675214-21sed22:15:220
2607851991910,7cyclictest436-20systemd-journal23:55:010
260785199190,1cyclictest0-21swapper/023:25:330
260785199190,18cyclictest0-21swapper/021:36:090
2607864991816,1cyclictest3056-21PK-Backend22:58:493
2607864991816,1cyclictest3056-21PK-Backend21:41:163
2607864991816,1cyclictest3056-21PK-Backend19:24:013
2607864991816,1cyclictest0-21swapper/323:36:423
2607864991815,2cyclictest3056-21PK-Backend21:46:293
260785199188,8cyclictest918-21rs:main0
2607851991816,1cyclictest3056-21PK-Backend00:19:330
2607851991816,1cyclictest3056-21PK-Backend00:19:330
2607851991816,1cyclictest0-21swapper/019:33:530
2607851991816,1cyclictest0-21swapper/000:27:510
2607851991816,1cyclictest0-21swapper/000:09:120
2607851991814,3cyclictest1095-21runrttasks23:30:450
2607851991810,7cyclictest2681166-21apt22:35:110
2607864991715,1cyclictest3056-21PK-Backend23:04:103
260785199179,6cyclictest2718557-21cat00:20:170
260785199179,6cyclictest2718557-21cat00:20:160
2607851991711,5cyclictest2698007-21sed23:20:220
2607851991711,5cyclictest2698007-21sed23:20:220
2607864991614,1cyclictest3056-21PK-Backend22:52:403
2607864991614,1cyclictest3056-21PK-Backend22:17:523
2607859991614,1cyclictest0-21swapper/219:46:362
260785199168,7cyclictest2635480-21cpuspeed_turbos20:25:120
260785199162,11cyclictest2702222-21apt23:35:110
2607851991614,1cyclictest0-21swapper/019:11:440
2607851991612,3cyclictest2670739-21/usr/sbin/munin22:05:190
260786499150,13cyclictest3056-21PK-Backend22:05:273
2607851991510,3cyclictest2659115-21kernelversion21:30:170
2607851991412,1cyclictest0-21swapper/000:36:100
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional