You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-20 - 12:11

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Thu Aug 20, 2026 00:43:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
294429659,10sleep30-21swapper/319:09:543
280228876,8sleep00-21swapper/019:08:020
281828359,13sleep10-21swapper/119:08:151
83028168,9sleep20-21swapper/219:05:192
304699300,29cyclictest0-21swapper/223:28:172
304699300,29cyclictest0-21swapper/222:03:422
304699300,23cyclictest1690-21sh22:50:142
304299300,28cyclictest0-21swapper/120:55:191
3039992929,0cyclictest0-21swapper/023:45:340
304699283,24cyclictest0-21swapper/200:10:402
304699280,27cyclictest0-21swapper/223:11:332
304699280,21cyclictest0-21swapper/221:55:142
303999282,2cyclictest22956-21munin-run00:20:000
303999280,0cyclictest0-21swapper/022:35:150
304699271,20cyclictest22954-21perf00:19:592
304699270,26cyclictest0-21swapper/222:31:002
304699270,26cyclictest0-21swapper/200:00:162
303999271,2cyclictest2400-21ssh22:50:170
303999271,16cyclictest0-21swapper/000:20:130
304699260,26cyclictest0-21swapper/222:48:232
304699260,25cyclictest0-21swapper/223:07:212
304699260,25cyclictest0-21swapper/222:25:122
304299260,23cyclictest0-21swapper/121:36:401
303999261,24cyclictest0-21swapper/023:15:150
303999260,25cyclictest0-21swapper/023:35:130
303999260,1cyclictest0-21swapper/022:25:120
303999260,1cyclictest0-21swapper/021:28:570
303999260,1cyclictest0-21swapper/020:00:150
304699255,19cyclictest12170-21tune2fs23:15:152
304699250,22cyclictest0-21swapper/222:35:152
3039992522,2cyclictest0-21swapper/023:41:260
303999251,23cyclictest0-21swapper/022:15:130
303999250,3cyclictest0-21swapper/022:55:140
303999250,3cyclictest0-21swapper/000:10:340
303999250,2cyclictest0-21swapper/000:29:070
304699247,13cyclictest0-21swapper/221:51:032
304699240,23cyclictest0-21swapper/223:57:472
304699240,19cyclictest0-21swapper/219:35:132
304699240,18cyclictest9118-21ssh00:30:222
303999243,19cyclictest28058-21diskmemload22:06:170
303999241,1cyclictest28058-21diskmemload21:51:020
304699231,21cyclictest28058-21diskmemload00:27:452
304699231,21cyclictest0-21swapper/221:25:592
304699231,13cyclictest18484-21apt-get23:00:132
304699230,21cyclictest25411-21latency_hist22:45:002
303999232,18cyclictest23119-21ssh21:45:130
303999230,4cyclictest28058-21diskmemload23:59:570
303999230,2cyclictest0-21swapper/019:40:110
303999230,1cyclictest0-21swapper/023:00:130
303999230,1cyclictest0-21swapper/000:30:120
303999230,17cyclictest0-21swapper/000:06:450
304699227,14cyclictest28058-21diskmemload23:51:222
304699222,19cyclictest28058-21diskmemload22:09:062
304699222,17cyclictest23117-21df_inode21:45:132
304699221,20cyclictest28058-21diskmemload00:06:202
304699220,20cyclictest0-21swapper/221:21:382
304699220,1cyclictest0-21swapper/220:40:132
303999224,17cyclictest28058-21diskmemload22:00:320
303999223,18cyclictest28058-21diskmemload23:22:450
303999223,17cyclictest0-21swapper/019:20:150
303999223,16cyclictest28058-21diskmemload00:00:540
303999222,15cyclictest0-21swapper/022:10:150
303999221,3cyclictest0-21swapper/022:45:230
303999220,2cyclictest0-21swapper/023:53:030
303999220,19cyclictest0-21swapper/021:39:530
303999220,17cyclictest0-21swapper/020:20:110
304699217,12cyclictest28058-21diskmemload22:19:582
304699213,17cyclictest0-21swapper/222:55:132
304699212,18cyclictest28058-21diskmemload23:36:072
304699211,19cyclictest0-21swapper/221:10:142
304699211,15cyclictest0-21swapper/221:39:542
304699210,18cyclictest0-21swapper/221:15:582
304699210,13cyclictest28058-21diskmemload00:20:032
304299212,11cyclictest0-21swapper/119:20:131
303999213,17cyclictest28058-21diskmemload23:09:350
303999211,2cyclictest19442-21fschecks_time20:50:150
303999210,2cyclictest0-21swapper/022:30:130
3047992017,2cyclictest6046-21kworker/3:000:37:323
304699202,18cyclictest0-21swapper/223:41:392
304699201,11cyclictest29972-21apt-get21:30:142
304699200,17cyclictest0-21swapper/220:00:152
303999201,1cyclictest14247-21apt-get21:40:130
304799190,3cyclictest0-21swapper/322:35:133
304699192,16cyclictest22510-21latency_hist19:50:002
304699192,16cyclictest0-21swapper/219:40:122
304699191,8cyclictest0-21swapper/222:11:502
304699191,18cyclictest28058-21diskmemload23:30:412
304699191,17cyclictest28058-21diskmemload21:43:432
304699190,18cyclictest29759-21ssh23:45:152
304699190,17cyclictest0-21swapper/219:20:152
304699190,14cyclictest0-21swapper/223:20:162
304299190,1cyclictest0-21swapper/121:32:541
303999191,2cyclictest28058-21diskmemload22:42:210
303999191,2cyclictest0-21swapper/021:20:180
303999191,14cyclictest0-21swapper/022:21:490
303999190,2cyclictest0-21swapper/023:12:170
303999190,1cyclictest0-21swapper/021:55:370
303999190,1cyclictest0-21swapper/021:10:160
304799185,2cyclictest28058-21diskmemload00:09:003
304799180,17cyclictest0-21swapper/321:59:553
304699187,8cyclictest4002-21taskset20:17:072
304699180,18cyclictest0-21swapper/222:22:082
304699180,17cyclictest0-21swapper/219:50:172
303999182,15cyclictest0-21swapper/023:25:150
303999181,1cyclictest32722-21cut20:10:170
303999181,15cyclictest0-21swapper/021:31:460
303999180,1cyclictest0-21swapper/021:17:040
303999180,1cyclictest0-21swapper/019:10:230
303999180,17cyclictest0-21swapper/000:35:350
304699174,12cyclictest0-21swapper/219:15:132
304699172,14cyclictest32720-21tune2fs20:10:172
304699172,13cyclictest22393-21grep20:55:222
304699170,16cyclictest0-21swapper/220:50:122
304699170,15cyclictest28058-21diskmemload00:35:352
3042991716,1cyclictest0-21swapper/100:10:211
303999171,1cyclictest0-21swapper/019:15:220
303999170,2cyclictest0-21swapper/023:30:410
303999170,16cyclictest0-21swapper/021:05:130
303999170,16cyclictest0-21swapper/019:25:140
3047991615,1cyclictest22886-21kworker/3:123:09:223
304699167,7cyclictest31479-21taskset20:05:292
304699160,10cyclictest0-21swapper/220:30:142
304299169,6cyclictest30303-21cat21:10:261
304299169,6cyclictest24574-21/usr/sbin/munin22:25:141
304299160,16cyclictest0-21swapper/122:05:131
304299160,16cyclictest0-21swapper/100:15:291
304299160,14cyclictest0-21swapper/121:26:241
3047991514,1cyclictest25396-21kworker/3:021:28:223
3047991514,1cyclictest22886-21kworker/3:123:45:323
3047991514,1cyclictest22886-21kworker/3:123:20:113
304799151,13cyclictest28058-21diskmemload22:50:523
304799150,15cyclictest0-21swapper/321:46:013
304799150,15cyclictest0-21swapper/300:16:513
304699155,9cyclictest13146-21/usr/sbin/munin19:30:132
304699150,13cyclictest3379-21basename19:10:232
304299154,9cyclictest0-21swapper/120:05:131
304299150,1cyclictest0-21swapper/123:23:081
304299150,15cyclictest0-21swapper/122:40:381
303999158,6cyclictest16274-21cat20:45:000
303999158,6cyclictest13521-21chrt20:37:080
303999157,7cyclictest15976-21tr19:35:170
303999156,5cyclictest452-21systemd-logind21:01:370
303999156,5cyclictest452-21systemd-logind21:01:370
303999152,12cyclictest0-21swapper/019:50:000
303999151,2cyclictest0-21swapper/020:18:020
303999150,14cyclictest0-21swapper/019:30:130
304799149,3cyclictest4938-21cut23:30:143
304799149,3cyclictest12292-21cut20:35:163
3047991413,1cyclictest6046-21kworker/3:000:33:323
3047991411,2cyclictest5027-21grep19:10:333
3047991410,3cyclictest32536-21awk22:30:003
304699149,4cyclictest10671-21awk19:25:142
304699148,5cyclictest6510-21taskset20:23:352
304699147,6cyclictest16704-21expr20:45:122
304699142,11cyclictest23850-21cstates21:00:152
304699142,11cyclictest23850-21cstates21:00:152
304699140,11cyclictest25383-21cstates19:55:132
303999148,5cyclictest3824-21apache220:30:110
303999147,6cyclictest31816-21chrt20:09:520
303999147,6cyclictest18327-21seq20:47:390
303999145,6cyclictest23267-21sh21:00:000
304799139,3cyclictest4659-21grep20:20:143
304799139,3cyclictest23870-21cat00:00:343
304799138,3cyclictest32218-21ssh22:10:163
304799134,6cyclictest0-21swapper/322:47:193
304799132,11cyclictest25396-21kworker/3:021:21:513
304799131,3cyclictest0-21swapper/321:18:113
3047991312,1cyclictest0-21swapper/323:03:133
304799131,10cyclictest0-21swapper/323:51:053
3047991310,2cyclictest931-21wc20:10:233
3047991310,2cyclictest4246-21turbostat.cron20:20:003
3047991310,2cyclictest16489-21wc19:35:213
3047991310,2cyclictest14371-21head23:55:173
3047991310,2cyclictest12801-21grep23:35:143
304799130,13cyclictest0-21swapper/322:09:163
304699139,3cyclictest278402sleep221:07:322
304699130,9cyclictest225-21systemd-journal20:30:002
304299139,3cyclictest7816-21cron19:20:001
304299139,3cyclictest7785-21head21:55:181
3042991312,1cyclictest55750irq/128-enp2s0-00:09:131
3042991310,2cyclictest31032-21wc23:45:261
3042991310,2cyclictest30014-21grep23:25:261
3042991310,2cyclictest24806-21wc19:55:001
3042991310,2cyclictest22762-21grep21:45:121
3042991310,2cyclictest11530-21ssh00:31:451
304299130,13cyclictest0-21swapper/122:22:031
304299130,12cyclictest16703-21ssh22:39:541
264332130,1sleep30-21swapper/300:21:133
304799129,2cyclictest8877-21tail19:20:173
304799129,2cyclictest8153-21wc20:25:223
304799129,2cyclictest5852-21tr19:15:143
304799129,2cyclictest31513-21df21:50:163
304799129,2cyclictest30248-21tr21:30:163
304799129,2cyclictest26678-21tr21:05:153
304799129,2cyclictest18325-21df19:40:143
304799129,2cyclictest12736-21seq21:39:163
304799129,2cyclictest10764-21grep19:25:153
304799129,2cyclictest10037-21grep20:30:153
304799128,2cyclictest23547-21wc19:50:173
304799128,2cyclictest20968-21sed19:45:163
304799125,5cyclictest1-21systemd20:50:223
304799121,7cyclictest0-21swapper/323:12:123
3047991211,1cyclictest19029-21kworker/3:222:21:523
304699120,11cyclictest12865-21grep20:35:212
304299129,2cyclictest8589-21grep22:15:151
304299129,2cyclictest5979-21grep23:30:231
304299129,2cyclictest4892-21grep21:15:141
304299129,2cyclictest32241-21grep20:10:151
304299129,2cyclictest2514-21tail22:50:171
304299129,2cyclictest2331-21tr20:15:151
304299129,2cyclictest20350-21tr19:45:141
304299129,2cyclictest19261-21grep23:00:171
304299129,2cyclictest17893-21cat20:45:241
304299129,2cyclictest14977-21sed23:55:221
304299129,2cyclictest13928-21chrt23:15:501
304299129,2cyclictest13345-21grep20:35:261
304299128,3cyclictest27539-21chrt00:03:151
304299127,4cyclictest20008-21rm21:24:191
304299123,5cyclictest225-21systemd-journal19:35:011
304299122,6cyclictest28058-21diskmemload00:30:011
304299120,9cyclictest0-21swapper/122:10:141
304299120,11cyclictest0-21swapper/120:40:131
303999129,2cyclictest26050-21cut19:55:170
303999120,4cyclictest225-21systemd-journal19:51:230
304799119,1cyclictest29161-21sed21:10:163
304799118,2cyclictest7451-21head00:10:173
304799118,2cyclictest4976-21sh22:32:143
304799118,2cyclictest32705-21cut00:25:173
304799118,2cyclictest31865-21ssh23:26:473
304799118,2cyclictest27634-21tr20:00:163
304799118,2cyclictest25728-21grep19:55:153
304799118,2cyclictest241312sleep321:00:163
304799118,2cyclictest241312sleep321:00:163
304799118,2cyclictest22860-21head20:55:273
304799118,2cyclictest21231-21grep23:40:163
304799118,2cyclictest20269-21ssh22:41:063
304799118,2cyclictest16706-21ssh22:00:223
304799118,2cyclictest14731-21grep20:40:163
304799118,2cyclictest14708-21ssh22:57:283
304799118,2cyclictest13267-21expr19:30:133
304799118,2cyclictest12319-21sh22:17:183
304799118,2cyclictest11269-21ssh23:15:073
304799115,5cyclictest18908-21rm21:42:323
304299118,2cyclictest9927-21ssh22:55:041
304299118,2cyclictest8728-21ssh23:51:341
304299118,2cyclictest4093-21sed19:10:261
304299118,2cyclictest31737-21head21:50:181
304299118,2cyclictest30586-21ssh22:47:411
304299118,2cyclictest29664-21cat20:05:001
304299118,2cyclictest28212-21date21:10:011
304299118,2cyclictest26606-21ssh23:43:201
304299118,2cyclictest25213-21tr19:55:121
304299118,2cyclictest19423-21cut20:50:151
304299118,2cyclictest1666-21ssh23:09:321
304299118,2cyclictest15061-21ssh22:00:041
304299118,2cyclictest1449-21grep22:30:181
304299118,2cyclictest10937-21sed20:30:251
304299116,4cyclictest24744-21ntpq00:20:201
304299116,1cyclictest16954-21grep00:35:161
304299111,3cyclictest225-21systemd-journal23:10:021
304299110,4cyclictest225-21systemd-journal19:45:001
303999114,6cyclictest6881-21bc20:25:120
304799108,1cyclictest31742-21seq20:08:563
304799107,2cyclictest16640-21grep20:45:133
304299108,1cyclictest15234-21apt-get19:35:131
304299108,1cyclictest13960-21sh21:40:011
304299107,2cyclictest4897-21expr20:20:131
304299107,2cyclictest14734-21ssh23:35:351
304299103,4cyclictest0-21swapper/121:00:171
304299103,4cyclictest0-21swapper/121:00:161
304299100,9cyclictest10625-21egrep19:25:141
30429997,1cyclictest7221-21expr20:25:151
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional