You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-11 - 08:05

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackcslot7s.osadl.org (updated Wed Mar 11, 2026 00:44:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
22416999468448,17cyclictest310927-21turbostat23:15:003
22415999464454,7cyclictest284926-21turbostat22:04:540
22416999463456,4cyclictest298780-21turbostat22:44:553
22416999463456,4cyclictest281470-21turbostat21:50:003
22416999463455,5cyclictest281470-21turbostat21:54:553
22416999462455,4cyclictest326834-21turbostat00:04:543
22415999462443,16cyclictest248799-21turbostat20:24:550
22416999461454,4cyclictest256031-21turbostat20:44:563
22416999460453,4cyclictest335502-21turbostat00:29:553
22416799460451,6cyclictest314406-21turbostat23:29:552
22416799457450,4cyclictest264980-21turbostat21:09:552
22416999455447,5cyclictest240149-21turbostat19:55:013
22416999454446,4cyclictest250532-21turbostat20:25:003
22416999454446,4cyclictest238419-21turbostat19:50:013
22416999453437,13cyclictest243601-21turbostat20:05:003
22416999453437,13cyclictest243601-21turbostat20:05:003
22416999452431,18cyclictest325102-21turbostat23:55:003
22416399451444,4cyclictest274247-21turbostat21:34:551
22416999449442,4cyclictest305693-21turbostat23:04:553
22416999448441,4cyclictest262943-21turbostat20:59:593
22416999448439,6cyclictest312668-21turbostat23:24:553
22416999447440,4cyclictest240149-21turbostat19:59:553
22416999447438,6cyclictest229761-21turbostat19:29:553
22415999445438,4cyclictest337232-21turbostat00:34:550
22416799443436,4cyclictest288388-21turbostat22:14:552
22416399443435,5cyclictest333769-21turbostat00:24:551
22416999442435,4cyclictest338963-21turbostat00:39:553
22416799442418,4cyclictest291850-21turbostat22:24:552
22416999441434,4cyclictest228025-21turbostat19:24:553
22416999438427,8cyclictest307425-21turbostat23:09:553
22416999437427,6cyclictest317875-21turbostat23:35:013
22416999437427,6cyclictest317875-21turbostat23:35:003
22416999435427,5cyclictest298780-21turbostat22:40:003
22416799434427,4cyclictest293572-21turbostat22:29:552
22416799434427,4cyclictest231489-21turbostat19:34:562
22416999432425,4cyclictest333769-21turbostat00:20:003
22416999429421,5cyclictest234944-21turbostat19:44:553
22416799425418,4cyclictest330295-21turbostat00:14:552
22416999424414,7cyclictest274247-21turbostat21:30:003
22415999424417,4cyclictest250532-21turbostat20:29:550
22416999423416,4cyclictest225811-21turbostat19:15:003
22416999422400,6cyclictest286654-21turbostat22:05:003
22416999421414,4cyclictest291850-21turbostat22:20:013
22416999421414,4cyclictest259493-21turbostat20:54:553
22416999420400,17cyclictest277706-21turbostat21:40:013
22416999420400,17cyclictest277706-21turbostat21:40:003
22416999418408,7cyclictest270474-21turbostat21:19:593
22416799416409,4cyclictest297046-21turbostat22:39:552
22416999413406,4cyclictest275971-21turbostat21:35:013
22416999413405,5cyclictest247066-21turbostat20:19:563
22416999413404,6cyclictest319607-21turbostat23:40:013
22416799412404,5cyclictest241873-21turbostat20:04:562
22416799412404,5cyclictest241873-21turbostat20:04:552
22415999412405,4cyclictest290124-21turbostat22:19:560
22416799405396,6cyclictest224029-21turbostat19:14:552
22416799403396,4cyclictest302226-21turbostat22:54:542
22416999401392,5cyclictest297046-21turbostat22:35:003
22416999398391,4cyclictest321334-21turbostat23:45:003
22416999396387,6cyclictest316128-21turbostat23:30:003
22416999395386,6cyclictest326834-21turbostat00:00:003
22416999395386,5cyclictest284926-21turbostat22:00:003
22416999394375,16cyclictest290124-21turbostat22:15:003
22416999392385,4cyclictest286654-21turbostat22:09:553
22416999391383,5cyclictest310927-21turbostat23:19:543
22416999391383,5cyclictest233229-21turbostat19:39:553
22416999391382,6cyclictest256031-21turbostat20:40:013
22416799391368,20cyclictest328581-21turbostat00:09:552
22416999388379,6cyclictest335502-21turbostat00:24:593
22416999386379,4cyclictest332042-21turbostat00:15:003
22416999386378,5cyclictest279438-21turbostat21:45:003
22416999384377,4cyclictest321334-21turbostat23:49:553
22416999384377,4cyclictest321334-21turbostat23:49:553
22416999382375,4cyclictest262943-21turbostat21:04:543
22415999379371,5cyclictest277706-21turbostat21:44:550
22416999376368,5cyclictest228025-21turbostat19:19:593
22416999374367,4cyclictest293572-21turbostat22:25:003
22416999374367,4cyclictest243601-21turbostat20:09:553
22416799367361,3cyclictest303966-21turbostat22:59:562
22416999366358,5cyclictest272505-21turbostat21:25:003
22416999365358,4cyclictest267014-21turbostat21:10:013
22416999363356,4cyclictest295308-21turbostat22:30:013
22416799360350,7cyclictest267014-21turbostat21:14:562
22416999359352,4cyclictest300506-21turbostat22:49:563
22416999357351,3cyclictest252252-21turbostat20:34:553
22416799357350,4cyclictest295308-21turbostat22:34:562
22416399355348,4cyclictest245338-21turbostat20:14:541
22415999353345,5cyclictest236683-21turbostat19:49:550
22415999351343,5cyclictest257756-21turbostat20:49:550
22416799349342,4cyclictest268739-21turbostat21:19:562
22415999349343,3cyclictest279438-21turbostat21:49:550
22416999328321,4cyclictest268739-21turbostat21:15:003
22416799324318,3cyclictest253994-21turbostat20:39:542
22416999322313,6cyclictest259493-21turbostat20:50:003
22416999317311,3cyclictest330295-21turbostat00:10:003
22416999317295,19cyclictest305693-21turbostat23:00:003
22416799317310,4cyclictest309161-21turbostat23:14:552
22274722820,11sleep20-21swapper/219:05:142
22416999214200,10cyclictest233229-21turbostat19:35:013
22416999165151,10cyclictest252252-21turbostat20:30:003
223767212372,41sleep30-21swapper/319:07:003
223987210775,23sleep10-21swapper/119:09:521
223811210776,22sleep00-21swapper/019:07:330
2241699910693,9cyclictest247066-21turbostat20:15:013
2241699910192,6cyclictest303966-21turbostat22:55:013
224159999992,4cyclictest275971-21turbostat21:39:560
224159999992,4cyclictest275971-21turbostat21:39:550
224159999988,8cyclictest272505-21turbostat21:29:560
197399990,32rtkit-daemon1081-21ntpd23:01:530
224169999790,4cyclictest338963-21turbostat00:35:003
224159999588,4cyclictest323067-21turbostat23:54:560
2706882700,5sleep0161rcuc/021:20:010
197399620,9rtkit-daemon1972-21rtkit-daemon22:21:490
224159996111,36cyclictest1137-21runrttasks19:24:140
197399610,9rtkit-daemon1972-21rtkit-daemon22:49:011
224167996021,33cyclictest282086-21df21:50:142
197399590,9rtkit-daemon1972-21rtkit-daemon23:18:211
197399590,9rtkit-daemon1972-21rtkit-daemon23:04:281
197399590,9rtkit-daemon1972-21rtkit-daemon22:54:571
197399580,8rtkit-daemon1972-21rtkit-daemon22:22:591
224159995713,24cyclictest1137-21runrttasks00:28:410
197399570,8rtkit-daemon1972-21rtkit-daemon22:25:531
197399560,9rtkit-daemon1972-21rtkit-daemon22:44:111
197399550,9rtkit-daemon1972-21rtkit-daemon22:32:220
197399550,8rtkit-daemon1972-21rtkit-daemon23:12:011
197399550,8rtkit-daemon1972-21rtkit-daemon22:55:541
22415999548,22cyclictest255937-21chrt20:39:090
224159995415,34cyclictest328571-21latency_hist00:05:000
224159995413,19cyclictest309118-21taskset23:09:470
224159995412,17cyclictest275244-21ntpq21:30:230
197399540,8rtkit-daemon1972-21rtkit-daemon23:23:231
224167995318,11cyclictest438-20systemd-journal00:34:552
22415999539,22cyclictest264754-40lsb_release21:02:300
22415999538,21cyclictest234601-21chrt19:35:410
224159995316,32cyclictest313073-21tr23:20:130
197399530,9rtkit-daemon1972-21rtkit-daemon00:18:332
224159995214,16cyclictest230027-21chrt19:25:110
224159995212,18cyclictest320016-21cpuspeed_turbos23:40:130
224167995120,8cyclictest339359-21tr00:35:152
224167995118,10cyclictest251461-21seq20:25:202
22415999519,15cyclictest287503-21memory22:05:210
22415999517,21cyclictest224405-21cpuspeed19:10:130
224159995116,31cyclictest236689-21perf19:44:590
224159995113,17cyclictest312413-21seq23:17:050
224167995021,25cyclictest259874-21cpuspeed_turbos20:50:152
224167995017,7cyclictest254359-21date20:35:012
224167995017,11cyclictest299247-21cstates22:40:122
22416799501,45cyclictest0-21swapper/223:40:212
22415999509,16cyclictest334786-21ntp_states00:20:260
22415999508,15cyclictest923-21dbus-daemon23:40:000
224167994918,6cyclictest235193-21/usr/sbin/munin19:40:302
224167994915,11cyclictest256638-21expr20:40:142
22415999499,19cyclictest268637-21chrt21:13:530
22415999499,18cyclictest295009-21chrt22:26:220
22415999498,18cyclictest253823-21chrt20:33:050
22415999497,19cyclictest267228-21cut21:10:030
224159994914,17cyclictest241152-21ntp_states19:55:230
224159994914,16cyclictest299037-21apt22:40:100
224159994910,19cyclictest1137-21runrttasks20:04:540
224159994910,19cyclictest1137-21runrttasks20:04:540
224167994815,8cyclictest274220-21cron21:34:552
224167994811,12cyclictest240782-21sh19:55:162
224159994810,18cyclictest1137-21runrttasks23:32:180
224159994810,18cyclictest1137-21runrttasks23:32:170
224167994719,7cyclictest306767-21processes23:00:252
224167994718,7cyclictest287403-21kernelversion22:05:202
224167994717,8cyclictest317051-21needreboot23:30:212
224167994717,8cyclictest317051-21needreboot23:30:212
22415999479,15cyclictest301890-21chrt22:45:510
22415999478,18cyclictest232938-21chrt19:31:310
22415999478,16cyclictest256290-21dpkg20:40:110
22415999478,15cyclictest340354-21chrt00:35:530
22415999477,15cyclictest246759-21taskset20:11:190
22415999477,15cyclictest244071-21cstates20:05:140
22415999476,25cyclictest0-21swapper/023:28:280
22415999472,16cyclictest0-21swapper/020:15:210
22416799463,8cyclictest0-21swapper/200:00:012
224167994619,8cyclictest280090-21tr21:45:122
22415999469,15cyclictest326713-21chrt23:58:470
224159994614,28cyclictest239229-21cat19:50:210
2806872450,6sleep10-21swapper/121:45:231
224167994518,7cyclictest1137-21runrttasks23:05:422
224167994516,7cyclictest1137-21runrttasks21:58:252
224167994515,7cyclictest301296-21latency22:45:192
224167994513,9cyclictest314443-21ls23:25:012
224163994515,7cyclictest3298022timerandwakeup00:05:301
22415999458,15cyclictest1-21systemd23:00:000
22415999457,14cyclictest0-21swapper/000:16:170
22416799443,18cyclictest325134-21grep23:55:012
224167994419,6cyclictest328572-21idleruntime-cro00:05:012
22416799441,5cyclictest141rcu_preempt23:50:002
22416799441,5cyclictest141rcu_preempt23:50:002
224167994413,11cyclictest247539-21cstates20:15:152
22415999448,15cyclictest298480-21chrt22:36:240
22415999447,15cyclictest310765-21chrt23:13:100
224159994413,27cyclictest284045-21/usr/sbin/munin21:55:220
224159994412,29cyclictest282437-21ntp_kernel_err21:50:240
224167994317,7cyclictest261647-21tr20:55:132
224167994316,7cyclictest4497-21PK-Backend19:19:012
224167994311,7cyclictest0-21swapper/223:38:262
224167994310,11cyclictest923-21dbus-daemon21:05:002
224163994324,15cyclictest324516-40lsb_release23:50:281
224163994324,14cyclictest26-21ksoftirqd/122:05:251
224163994321,18cyclictest316708-21cstates23:30:131
224163994321,18cyclictest316708-21cstates23:30:131
22415999436,13cyclictest0-21swapper/022:51:520
22415999436,13cyclictest0-21swapper/000:13:060
224167994214,8cyclictest4497-21PK-Backend19:53:222
22416799421,13cyclictest244721-21ps20:05:262
224167994210,11cyclictest976-21rs:main2
22416399427,30cyclictest339677-21iwlist00:35:211
224163994224,6cyclictest26-21ksoftirqd/122:30:281
224163994224,6cyclictest26-21ksoftirqd/121:50:211
224163994224,6cyclictest26-21ksoftirqd/120:27:341
224163994223,6cyclictest26-21ksoftirqd/120:45:291
224163994223,6cyclictest26-21ksoftirqd/119:32:191
22415999427,13cyclictest0-21swapper/021:23:190
197399420,33rtkit-daemon4497-21PK-Backend00:25:032
224167994114,9cyclictest356-21jbd2/sdb2-821:20:312
224163994124,6cyclictest26-21ksoftirqd/119:30:001
224163994123,6cyclictest26-21ksoftirqd/123:48:451
224163994123,6cyclictest26-21ksoftirqd/123:48:441
224163994123,6cyclictest26-21ksoftirqd/120:18:441
224163994123,6cyclictest26-21ksoftirqd/120:05:231
224163994123,6cyclictest26-21ksoftirqd/119:55:511
224163994123,6cyclictest26-21ksoftirqd/119:40:121
224163994121,7cyclictest26-21ksoftirqd/123:05:261
22415999417,13cyclictest0-21swapper/023:48:440
22415999417,13cyclictest0-21swapper/023:48:440
22415999414,14cyclictest0-21swapper/000:09:550
2852352400,6sleep234-21ksoftirqd/222:00:132
224163994023,6cyclictest26-21ksoftirqd/122:10:121
224163994023,6cyclictest26-21ksoftirqd/119:16:101
224163994023,5cyclictest26-21ksoftirqd/122:35:261
224163994023,5cyclictest26-21ksoftirqd/121:43:031
224163994023,5cyclictest26-21ksoftirqd/120:57:051
224163994023,5cyclictest26-21ksoftirqd/120:41:171
224163994023,5cyclictest26-21ksoftirqd/120:33:391
224163994022,6cyclictest26-21ksoftirqd/100:00:231
224163994022,5cyclictest26-21ksoftirqd/120:35:071
22415999409,14cyclictest1137-21runrttasks20:54:050
224159994017,19cyclictest225845-21munin-node19:15:060
197399400,27rtkit-daemon4497-21PK-Backend21:26:442
197399400,27rtkit-daemon4497-21PK-Backend19:25:282
22416799395,15cyclictest234234-21ntp_states19:35:262
22416799392,8cyclictest290742-21df22:15:152
224167993917,6cyclictest250530-21/usr/sbin/munin20:25:012
22416799391,17cyclictest259481-21latency_hist20:50:022
224163993923,6cyclictest26-21ksoftirqd/100:30:111
224163993922,5cyclictest26-21ksoftirqd/123:35:301
224163993922,5cyclictest26-21ksoftirqd/123:25:391
224163993922,5cyclictest26-21ksoftirqd/119:48:011
224163993921,6cyclictest26-21ksoftirqd/121:05:281
224163993921,6cyclictest26-21ksoftirqd/100:16:181
224163993920,6cyclictest26-21ksoftirqd/120:50:191
22416399391,17cyclictest2285292ls19:20:141
22416799381,17cyclictest237431-21irqstats19:45:192
224163993822,5cyclictest26-21ksoftirqd/120:20:291
224163993822,5cyclictest26-21ksoftirqd/119:35:271
224163993821,6cyclictest26-21ksoftirqd/121:37:401
224163993821,6cyclictest26-21ksoftirqd/121:37:401
224163993821,5cyclictest26-21ksoftirqd/121:58:541
224163993821,5cyclictest26-21ksoftirqd/121:15:151
224163993820,6cyclictest26-21ksoftirqd/123:55:241
224163993820,6cyclictest26-21ksoftirqd/121:20:071
224163993820,6cyclictest26-21ksoftirqd/119:54:501
224163993810,12cyclictest337110-21seq00:28:401
22415999388,26cyclictest289941-21seq22:13:010
224159993814,20cyclictest262753-21chrt20:57:430
224167993720,13cyclictest311940-21ntpq23:15:252
224163993722,11cyclictest242517-21sshd20:00:181
224163993722,11cyclictest242517-21sshd20:00:171
224163993720,5cyclictest26-21ksoftirqd/122:18:461
224163993720,5cyclictest26-21ksoftirqd/121:11:441
224163993720,5cyclictest26-21ksoftirqd/121:00:101
224163993719,13cyclictest286271-21users22:00:291
224163993518,5cyclictest26-21ksoftirqd/121:27:431
224163993518,5cyclictest26-21ksoftirqd/100:10:071
22416799341,30cyclictest0-21swapper/219:23:352
22416399346,13cyclictest4497-21PK-Backend23:40:161
224163993419,5cyclictest26-21ksoftirqd/119:12:321
224167993327,4cyclictest0-21swapper/221:42:082
224167993327,4cyclictest0-21swapper/220:13:182
22416799311,4cyclictest0-21swapper/221:37:452
22416799311,4cyclictest0-21swapper/221:37:442
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional