You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-11 - 15:03

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Sat Jul 11, 2026 12:43:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
327029985,9sleep00-21swapper/007:07:100
320128961,9sleep30-21swapper/307:06:163
342628875,9sleep20-21swapper/207:09:122
135227858,6sleep10-21swapper/107:05:061
82692670,0sleep20-21swapper/209:16:022
200622620,0sleep30-21swapper/312:37:113
267502580,0sleep00-21swapper/009:27:260
310272290,0sleep20-21swapper/210:47:502
3583992826,1cyclictest16058-21pmu-power12:15:223
357899275,21cyclictest24459-21awk12:20:221
3576992727,0cyclictest0-21swapper/009:49:510
3583992625,1cyclictest0-21swapper/310:08:553
3576992620,5cyclictest13-21ksoftirqd/010:04:410
358199240,24cyclictest0-21swapper/212:26:392
358199240,24cyclictest0-21swapper/210:32:152
358199240,0cyclictest0-21swapper/211:53:372
357699246,8cyclictest13-21ksoftirqd/011:14:340
3583992317,5cyclictest0-21swapper/310:59:513
357899230,19cyclictest0-21swapper/109:30:041
3576992319,3cyclictest21852-21cron08:55:010
358399221,19cyclictest22166-21sh10:42:273
3576992216,1cyclictest0-21swapper/012:22:310
358399210,19cyclictest26406-21kworker/3:111:00:033
358199217,7cyclictest0-21swapper/210:20:112
3581992120,1cyclictest0-21swapper/209:52:472
358199210,21cyclictest0-21swapper/210:18:102
358199210,21cyclictest0-21swapper/210:18:102
358199210,21cyclictest0-21swapper/209:28:332
3578992120,1cyclictest25-21ksoftirqd/110:18:501
3578992120,1cyclictest25-21ksoftirqd/110:18:501
357899210,20cyclictest0-21swapper/110:51:211
357899210,1cyclictest0-21swapper/112:14:341
357899210,1cyclictest0-21swapper/112:14:341
357899210,0cyclictest0-21swapper/109:22:091
3576992110,11cyclictest0-21swapper/010:39:140
358399200,19cyclictest26406-21kworker/3:110:54:293
3581992018,1cyclictest0-21swapper/209:44:232
358199200,1cyclictest0-21swapper/211:15:452
3578992018,1cyclictest19751-21apt-get08:50:131
357899201,3cyclictest0-21swapper/112:05:141
357899200,0cyclictest0-21swapper/109:28:391
3576992015,3cyclictest13-21ksoftirqd/010:50:210
358399190,18cyclictest0-21swapper/310:45:343
3578991917,1cyclictest13894-21sed11:35:201
357899191,2cyclictest0-21swapper/109:53:221
3576991915,3cyclictest0-21swapper/012:35:160
3576991911,6cyclictest29010-21diskmemload09:23:240
358399180,17cyclictest0-21swapper/309:32:513
358199180,1cyclictest0-21swapper/211:34:422
358199180,18cyclictest0-21swapper/212:38:392
358199180,18cyclictest0-21swapper/212:31:392
358199180,18cyclictest0-21swapper/211:47:572
358199180,18cyclictest0-21swapper/210:51:432
358199180,18cyclictest0-21swapper/209:38:542
357899189,8cyclictest537-21snmpd11:48:511
3578991816,1cyclictest27871-21latency09:05:191
357899180,18cyclictest0-21swapper/111:30:151
357899180,17cyclictest0-21swapper/112:17:071
357899180,0cyclictest0-21swapper/111:51:461
357699184,11cyclictest13-21ksoftirqd/010:33:400
3576991816,1cyclictest4462-21taskset09:33:520
3576991813,4cyclictest29010-21diskmemload10:56:030
358399170,0cyclictest0-21swapper/310:38:123
3581991716,1cyclictest0-21swapper/212:15:522
3581991716,1cyclictest0-21swapper/210:05:332
358199170,17cyclictest0-21swapper/212:05:222
358199170,17cyclictest0-21swapper/209:59:032
358199170,17cyclictest0-21swapper/209:20:232
358199170,16cyclictest0-21swapper/212:01:472
3578991716,1cyclictest0-21swapper/111:15:351
357699173,11cyclictest12674-21/usr/sbin/munin11:35:190
3576991715,1cyclictest25973-21apt-get10:45:140
3576991713,3cyclictest6844-21idleruntime-cro12:30:010
3576991713,3cyclictest13-21ksoftirqd/009:12:410
358399169,6cyclictest5926-21apache212:04:523
358399160,1cyclictest0-21swapper/311:34:463
358399160,16cyclictest0-21swapper/311:50:343
358199160,2cyclictest0-21swapper/212:12:502
358199160,2cyclictest0-21swapper/212:12:502
357899162,10cyclictest23713-21awk12:40:011
357699168,1cyclictest29010-21diskmemload12:08:460
357699168,1cyclictest141rcu_preempt10:19:360
357699168,1cyclictest141rcu_preempt10:19:350
357699167,5cyclictest29010-21diskmemload11:17:460
357699160,1cyclictest0-21swapper/010:44:480
358399156,9cyclictest5914-21apache210:17:023
358399156,9cyclictest5914-21apache210:17:023
358399154,7cyclictest5931-21cups-browsed08:30:013
358399154,4cyclictest0-21swapper/312:06:303
3583991514,1cyclictest0-21swapper/310:12:253
3581991514,1cyclictest0-21swapper/212:22:032
3581991514,1cyclictest0-21swapper/211:43:022
3581991514,1cyclictest0-21swapper/211:09:232
3581991514,1cyclictest0-21swapper/210:55:232
3581991510,3cyclictest3139-21df08:15:152
358199150,15cyclictest0-21swapper/211:24:152
3578991514,1cyclictest0-21swapper/112:00:111
3578991512,3cyclictest25-21ksoftirqd/112:26:221
3578991510,2cyclictest0-21swapper/107:50:141
357699157,2cyclictest0-21swapper/010:12:450
357699153,7cyclictest29912-21perf12:05:000
357699152,7cyclictest13145-21idleruntime-cro07:30:000
357699152,13cyclictest13-21ksoftirqd/011:05:030
357699151,14cyclictest0-21swapper/012:10:150
357699151,14cyclictest0-21swapper/012:10:150
3576991510,1cyclictest7177-21perf08:25:010
3576991510,1cyclictest29010-21diskmemload11:49:230
358399148,4cyclictest21101-21apache_accesses11:40:143
3583991410,3cyclictest8564-21tr09:55:203
3583991410,3cyclictest22490-21tr09:25:163
358399140,1cyclictest0-21swapper/312:30:373
358199145,5cyclictest225-21systemd-journal10:40:012
358199141,1cyclictest0-21swapper/210:42:042
3581991410,3cyclictest3611-21ls11:29:592
3581991410,3cyclictest22816-21tune2fs08:55:182
357899149,4cyclictest22501-21cron09:45:001
357899142,1cyclictest141rcu_preempt10:12:041
357899142,1cyclictest141rcu_preempt07:40:151
3578991413,1cyclictest0-21swapper/110:01:001
3578991412,2cyclictest0-21swapper/110:45:431
357899141,12cyclictest0-21swapper/110:59:431
357899140,14cyclictest0-21swapper/110:05:241
357699148,6cyclictest0-21swapper/009:57:200
357699148,5cyclictest0-21swapper/011:42:200
357699147,1cyclictest29010-21diskmemload10:22:120
357699146,7cyclictest27703-21munin-run11:25:000
357699144,5cyclictest225-21systemd-journal07:24:590
358399139,3cyclictest6398-21sed09:15:173
358399139,3cyclictest3591-21cron11:29:593
358399139,3cyclictest31406-21grep09:50:143
3583991311,1cyclictest27907-21grep08:00:143
3583991310,2cyclictest4044-21cat08:15:253
3583991310,2cyclictest28993-21cron09:09:593
3583991310,2cyclictest19780-21grep08:50:123
3583991310,2cyclictest17723-21ssh09:41:143
3583991310,2cyclictest14130-21grep07:30:183
358399130,12cyclictest9771-21ls08:30:003
358399130,10cyclictest0-21swapper/311:47:403
358199134,7cyclictest13965-21cpuspeed_turbos11:55:162
3581991310,2cyclictest8029-21grep08:25:152
3581991310,2cyclictest21158-21cut07:45:182
358199130,0cyclictest0-21swapper/211:38:052
357899139,3cyclictest15995-21tr07:35:161
357899133,5cyclictest141rcu_preempt10:30:281
3578991310,3cyclictest38150irq/131-snd_hda09:38:131
3578991310,2cyclictest7493-21grep09:15:281
3578991310,2cyclictest16021-21tr08:40:201
3578991310,2cyclictest14875-21wc07:30:281
3578991310,1cyclictest29010-21diskmemload12:30:181
357899130,13cyclictest0-21swapper/109:10:321
357699139,3cyclictest16307-21sh12:15:240
357699138,4cyclictest0-21swapper/011:30:000
357699138,1cyclictest29010-21diskmemload09:18:450
357699135,5cyclictest0-21swapper/009:51:120
357699134,7cyclictest15697-21kworker/0:209:37:350
3576991312,1cyclictest0-21swapper/010:05:450
3576991312,1cyclictest0-21swapper/009:40:160
3576991310,2cyclictest161rcuc/011:00:110
358399129,2cyclictest7194-21grep08:25:013
358399129,2cyclictest31812-21ssh10:29:183
358399129,2cyclictest30348-21grep09:10:183
358399129,2cyclictest22191-21sed07:45:293
358399129,2cyclictest18413-21grep07:40:153
358399129,2cyclictest14630-21sed11:35:273
358399129,2cyclictest14467-21grep11:55:183
358399128,2cyclictest7989-21wc09:35:233
358399127,4cyclictest27684-21sh11:25:003
3583991210,1cyclictest23151-21grep07:50:153
358399120,7cyclictest18684-21ssh09:22:393
358399120,11cyclictest24335-21grep09:45:213
358399120,11cyclictest12366-21grep11:15:193
358199129,2cyclictest5972-21idleruntime-cro07:15:002
358199129,2cyclictest5972-21apt09:34:592
358199129,2cyclictest32501-21ssh10:10:152
358199129,2cyclictest30781-21chrt09:10:212
358199129,2cyclictest26988-21head07:55:282
358199129,2cyclictest24795-21tr09:00:152
358199129,2cyclictest21371-21grep08:50:252
358199129,2cyclictest20809-21ssh11:00:262
358199129,2cyclictest19500-21awk08:50:012
358199129,2cyclictest16543-21grep07:35:182
358199129,2cyclictest10119-21tr08:30:142
358199128,3cyclictest3410-21tr11:10:152
357899129,3cyclictest0-21swapper/111:04:101
357899129,2cyclictest8115-21grep09:55:181
357899129,2cyclictest6849-21sort07:15:181
357899129,2cyclictest28091-21grep11:05:181
357899129,2cyclictest16705-21tr10:20:121
357899129,2cyclictest10289-21grep08:30:141
357899128,3cyclictest25237-21tail09:00:171
357899128,3cyclictest21985-21wc07:45:281
357899128,2cyclictest29089-21wc11:44:591
3578991210,0cyclictest0-21swapper/110:37:511
357899120,11cyclictest23751-21sort09:45:181
357699129,2cyclictest6478-21tr07:15:160
357699129,2cyclictest24997-21grep10:25:150
357699129,2cyclictest23211-21grep08:55:200
357699129,2cyclictest23165-21tr07:50:150
357699126,5cyclictest28854-21ssh11:25:170
357699126,5cyclictest0-21swapper/012:33:250
357699124,7cyclictest29010-21diskmemload11:50:150
358399119,1cyclictest1085-21grep08:10:193
358399118,2cyclictest7021-21head07:15:193
358399118,2cyclictest32159-21tail12:25:163
358399118,2cyclictest31745-21sed08:05:283
358399118,2cyclictest3062-21ssh11:10:103
358399118,2cyclictest27666-21ssh12:22:313
358399118,2cyclictest27179-21/usr/sbin/munin11:05:233
358399118,2cyclictest26064-21head07:55:183
358399118,2cyclictest19377-21ssh10:01:513
358399118,2cyclictest16786-21ls10:20:133
358399118,2cyclictest16444-21sort07:35:183
358399118,2cyclictest13035-21cut08:35:153
358399118,2cyclictest11285-21ssh12:12:463
358399118,2cyclictest11285-21ssh12:12:453
358399118,2cyclictest11214-21cat07:25:143
358399116,2cyclictest225-21systemd-journal09:05:003
358399110,1cyclictest0-21swapper/310:32:133
358399110,10cyclictest17034-21cat08:45:013
358199118,2cyclictest9268-21sort07:20:162
358199118,2cyclictest6669-21grep08:20:262
358199118,2cyclictest28249-21cut09:05:222
358199118,2cyclictest26789-21grep10:25:272
358199118,2cyclictest24352-21ssh09:45:222
358199118,2cyclictest23619-21grep07:50:182
358199118,2cyclictest19932-21chrt07:40:502
358199118,2cyclictest17623-21ssh10:00:282
358199118,2cyclictest16439-21grep08:40:262
358199118,2cyclictest13783-21expr07:30:152
358199118,2cyclictest11219-21grep07:25:142
358199113,4cyclictest225-21systemd-journal08:04:592
357899118,2cyclictest9555-21dpkg08:30:001
357899118,2cyclictest4220-21sed08:15:261
357899118,2cyclictest31190-21ntpq08:05:221
357899118,2cyclictest29429-21needreboot11:25:201
357899118,2cyclictest28650-21sh10:26:491
357899118,2cyclictest20645-21ssh11:20:171
357899118,2cyclictest17836-21wc10:40:151
357899118,2cyclictest16129-21ssh11:55:461
357899118,2cyclictest11226-21sh11:15:001
357899118,2cyclictest10733-21taskset07:24:561
357899117,3cyclictest5046-21head07:10:231
357699118,2cyclictest3372-21head08:15:160
357699118,2cyclictest31268-21ls08:05:230
357699118,2cyclictest1995-21taskset08:13:380
357699118,2cyclictest16937-21cat08:45:000
357699118,2cyclictest16075-21cat07:35:150
357699118,2cyclictest13110-21grep08:35:170
357699115,4cyclictest13-21ksoftirqd/011:57:140
357699112,8cyclictest0-21swapper/009:05:130
357699112,6cyclictest13-21ksoftirqd/008:30:220
358399108,1cyclictest22440-21cat08:55:163
358399108,1cyclictest17683-21cat08:45:173
358399107,2cyclictest4427-21tune2fs07:10:173
358199108,1cyclictest19542sleep208:13:062
358199107,2cyclictest7904-21grep07:15:292
358199107,2cyclictest30832-21head08:05:172
358199107,2cyclictest12425-21cut08:35:132
357899108,1cyclictest27676-21chrt08:00:111
357899108,1cyclictest143382sleep108:38:061
357899107,2cyclictest26635-21cut07:55:241
357899107,2cyclictest22436-21wc08:55:161
357899107,2cyclictest18915-21wc08:45:281
357899107,2cyclictest11347-21ls07:25:141
357899100,9cyclictest5742-21tr08:20:171
357699108,1cyclictest28106-21expr08:00:130
357699108,1cyclictest18372-21cat07:40:150
357699107,2cyclictest8142-21grep08:25:170
357699107,2cyclictest26386-21taskset09:02:030
357699107,2cyclictest26242-21wc07:55:210
357699107,2cyclictest21439-21if_err_enp2s007:45:210
357699101,6cyclictest13-21ksoftirqd/007:30:220
357699101,6cyclictest13-21ksoftirqd/007:10:130
357699100,6cyclictest13-21ksoftirqd/008:45:220
35839996,2cyclictest10468-21chrt07:21:313
35789997,1cyclictest758-21tail08:10:171
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional