You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-24 - 13:44

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Sun May 24, 2026 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2992629379,9sleep00-21swapper/019:09:540
2982629177,10sleep20-21swapper/219:08:362
2967928867,6sleep10-21swapper/119:06:421
2990428366,13sleep30-21swapper/319:09:383
3002399331,31cyclictest31996-21seq23:30:150
3002399290,28cyclictest0-21swapper/021:46:430
3002399290,28cyclictest0-21swapper/021:46:420
3002399290,27cyclictest0-21swapper/023:20:010
3002999280,1cyclictest0-21swapper/200:34:542
3002399270,23cyclictest0-21swapper/022:41:400
3002999260,26cyclictest0-21swapper/221:15:382
3002399264,21cyclictest23035-21diskmemload00:11:370
3002399260,25cyclictest9787-21perf22:00:010
3002999250,25cyclictest0-21swapper/221:23:572
30025992521,2cyclictest25-21ksoftirqd/121:10:121
3002399250,19cyclictest0-21swapper/022:15:140
3003199240,23cyclictest0-21swapper/322:21:163
30025992413,2cyclictest25-21ksoftirqd/121:33:571
3002399245,17cyclictest23035-21diskmemload23:09:430
3002399242,21cyclictest0-21swapper/020:35:130
3002399242,19cyclictest23035-21diskmemload22:51:530
3002399241,22cyclictest23035-21diskmemload21:33:320
3002399240,23cyclictest0-21swapper/023:50:140
3002399240,22cyclictest0-21swapper/022:25:200
3002399231,20cyclictest0-21swapper/023:10:130
3002399231,15cyclictest17293-21/usr/sbin/munin21:25:180
3002399230,20cyclictest14985-21sed19:45:160
30025992221,0cyclictest0-21swapper/121:54:191
30025992217,2cyclictest25-21ksoftirqd/100:02:241
3002599220,22cyclictest0-21swapper/123:52:361
3002399222,19cyclictest19135-21ssh22:05:160
30023992210,11cyclictest23035-21diskmemload00:19:550
3002399220,21cyclictest23035-21diskmemload21:37:160
3002399220,21cyclictest0-21swapper/000:03:260
30031992120,1cyclictest0-21swapper/322:16:233
30025992118,1cyclictest25-21ksoftirqd/122:57:521
3002599210,1cyclictest17056-21ntp_kernel_pll_20:55:201
3002599210,1cyclictest0-21swapper/121:24:171
3002399211,20cyclictest23035-21diskmemload21:51:450
3002399211,16cyclictest23035-21diskmemload23:23:310
3002399211,15cyclictest0-21swapper/020:05:120
3002399210,20cyclictest23035-21diskmemload23:25:050
3002399210,20cyclictest0-21swapper/022:21:520
3002399210,19cyclictest0-21swapper/022:13:040
3002399210,18cyclictest0-21swapper/022:55:140
30031992020,0cyclictest0-21swapper/322:00:323
3003199200,3cyclictest0-21swapper/323:40:023
3002599203,16cyclictest0-21swapper/123:30:151
30025992018,1cyclictest25-21ksoftirqd/100:35:031
30025992018,1cyclictest241rcuc/121:19:211
30025992017,1cyclictest25-21ksoftirqd/121:44:101
3002399206,12cyclictest23035-21diskmemload00:34:310
3002399203,16cyclictest0-21swapper/023:55:140
3002399200,19cyclictest0-21swapper/022:36:470
3002399200,17cyclictest28148-21cut00:25:170
3002399200,17cyclictest0-21swapper/020:15:210
3002999194,9cyclictest1221-21/usr/sbin/munin21:35:152
3002999192,1cyclictest30988-21kworker/u8:200:14:302
3002599199,9cyclictest25-21ksoftirqd/119:40:141
3002599191,4cyclictest241rcuc/122:26:301
3002599190,19cyclictest0-21swapper/121:49:421
3002599190,19cyclictest0-21swapper/121:49:411
3002399194,13cyclictest23035-21diskmemload22:48:250
3002399190,18cyclictest0-21swapper/021:21:170
3002399190,18cyclictest0-21swapper/021:00:110
3002399190,18cyclictest0-21swapper/000:35:500
3002399190,17cyclictest23035-21diskmemload23:38:180
3002399190,15cyclictest0-21swapper/023:42:180
30031991818,0cyclictest0-21swapper/323:36:113
3002999181,14cyclictest0-21swapper/223:04:082
3002999180,1cyclictest0-21swapper/222:01:512
3002599187,5cyclictest25-21ksoftirqd/120:00:121
3002599181,6cyclictest7347-21ssh00:31:571
3002599181,1cyclictest141rcu_preempt00:24:491
3002599180,4cyclictest0-21swapper/100:06:561
3002599180,1cyclictest0-21swapper/122:04:091
3002399180,18cyclictest0-21swapper/022:03:220
3002399180,17cyclictest0-21swapper/021:15:150
3002399180,17cyclictest0-21swapper/000:05:030
3002399180,16cyclictest0-21swapper/023:47:120
3002399180,16cyclictest0-21swapper/021:42:130
3002399180,14cyclictest0-21swapper/022:30:030
3002999170,13cyclictest0-21swapper/223:55:222
3002599177,8cyclictest25-21ksoftirqd/123:07:491
3002599174,8cyclictest25-21ksoftirqd/123:42:251
3002599171,6cyclictest25-21ksoftirqd/123:27:151
30025991716,1cyclictest0-21swapper/122:40:241
30025991712,4cyclictest25-21ksoftirqd/122:06:431
3002599170,3cyclictest141rcu_preempt00:15:501
3002399171,15cyclictest6644-21grep19:25:250
3002399171,14cyclictest23035-21diskmemload23:02:080
30023991710,6cyclictest19532-21tr19:55:150
3002399170,14cyclictest0-21swapper/020:10:250
3002399170,14cyclictest0-21swapper/019:20:130
3003199161,1cyclictest27441-21apt-get00:25:143
30031991611,4cyclictest13600-21cat23:00:013
3003199160,1cyclictest0-21swapper/321:57:323
3003199160,1cyclictest0-21swapper/321:29:523
3002999160,2cyclictest0-21swapper/221:25:202
3002999160,16cyclictest23035-21diskmemload23:53:112
3002999160,14cyclictest0-21swapper/222:10:352
3002599167,2cyclictest141rcu_preempt21:26:211
3002599165,5cyclictest25-21ksoftirqd/123:37:121
3002599164,3cyclictest141rcu_preempt23:45:491
3002599164,1cyclictest141rcu_preempt22:35:271
30025991615,1cyclictest241rcuc/123:57:241
30025991614,2cyclictest25-21ksoftirqd/123:00:231
3002599160,3cyclictest0-21swapper/123:23:211
3002599160,2cyclictest141rcu_preempt21:36:231
3002399161,14cyclictest31876-21sed19:10:250
3002399161,14cyclictest23035-21diskmemload00:22:590
3002399161,12cyclictest15036-21grep20:50:260
3002399160,15cyclictest0-21swapper/019:30:140
3002399160,13cyclictest0-21swapper/019:40:140
3002399160,12cyclictest0-21swapper/019:50:140
30031991512,2cyclictest39-21ksoftirqd/321:48:373
30031991512,2cyclictest39-21ksoftirqd/321:48:363
30031991510,4cyclictest372-21cut23:30:193
30029991511,2cyclictest16157-21grep19:45:272
3002999150,15cyclictest0-21swapper/222:43:412
3002999150,13cyclictest0-21swapper/221:44:592
30025991514,1cyclictest0-21swapper/122:20:011
3002399150,9cyclictest0-21swapper/021:10:030
3002399150,14cyclictest0-21swapper/020:30:160
3002399150,11cyclictest0-21swapper/020:25:140
3003199148,4cyclictest9503-21ls21:20:243
30031991411,2cyclictest22371-21grep20:00:173
30031991410,2cyclictest15616-21tr19:45:213
3003199140,14cyclictest0-21swapper/322:54:403
3002999141,5cyclictest4421-21sh23:13:382
3002999141,1cyclictest141rcu_preempt22:55:412
30029991411,2cyclictest19359-21df22:05:172
3002999141,11cyclictest0-21swapper/223:34:302
30029991410,3cyclictest17730-21grep23:40:212
3002999140,14cyclictest0-21swapper/222:46:142
3002999140,10cyclictest0-21swapper/222:29:492
3002999140,0cyclictest0-21swapper/221:12:172
3002599144,2cyclictest141rcu_preempt23:18:371
3002599141,1cyclictest141rcu_preempt23:12:301
30025991410,3cyclictest9780-21munin-run22:00:001
3002399148,5cyclictest2190-21taskset19:18:270
3002399148,5cyclictest21709-21sh21:05:180
30023991410,3cyclictest12664-21wc20:45:260
3003199139,3cyclictest7283-21cron21:20:003
3003199139,2cyclictest31110-21wc20:20:013
30031991310,2cyclictest25850-21cat21:30:193
30031991310,2cyclictest21888-21ls23:05:003
30031991310,2cyclictest12256-21sort00:15:273
3002999139,3cyclictest24692-21cat23:25:182
3002999139,2cyclictest15409-21grep23:20:152
30029991311,1cyclictest21620-21kworker/u8:022:20:122
30029991310,2cyclictest3900-21ls22:34:592
30029991310,2cyclictest3693-21grep20:30:132
30029991310,2cyclictest28257-21grep00:25:182
30029991310,2cyclictest27769-21fgrep20:10:212
30029991310,2cyclictest2714-21tr19:20:142
30029991310,2cyclictest26933-21latency00:05:192
30029991310,2cyclictest19035-21grep21:00:152
30029991310,2cyclictest18561-21cat00:19:592
3002999130,12cyclictest0-21swapper/223:35:052
3002999130,0cyclictest0-21swapper/223:05:242
3002599139,3cyclictest22001-21grep21:05:211
3002599135,4cyclictest241rcuc/122:32:371
3002599135,2cyclictest25-21ksoftirqd/100:29:291
30025991310,2cyclictest2758-21grep00:10:181
30025991310,2cyclictest23209-21grep22:45:281
30025991310,2cyclictest10453-21tail19:35:171
30025991310,1cyclictest141rcu_preempt19:54:131
3002599130,12cyclictest29791-21cut22:50:141
3002399139,3cyclictest23179-21wc20:00:270
3002399137,5cyclictest803-21chrt20:23:510
3002399137,5cyclictest11650-21chrt19:37:470
3003199129,2cyclictest8769-21grep20:40:143
3003199129,2cyclictest2746-21grep19:20:143
3003199129,2cyclictest27096-21grep20:10:163
3003199129,2cyclictest23202-21grep23:05:163
3003199129,2cyclictest18368-21hwlatdetect00:00:183
3003199129,2cyclictest15919-21cut20:55:123
3003199129,2cyclictest13239-21wc00:35:243
3003199129,2cyclictest1304-21tune2fs21:35:163
3003199129,2cyclictest12375-21tr19:40:163
3003199128,3cyclictest2644-21cut00:10:183
3003199128,3cyclictest25447-21grep21:10:253
3003199126,2cyclictest225-21systemd-journal20:00:013
3003199120,11cyclictest0-21swapper/323:15:313
3002999129,3cyclictest0-21swapper/221:51:402
3002999129,2cyclictest30668-21irqstats22:50:182
3002999129,2cyclictest23366-21grep20:00:272
3002999129,2cyclictest19342-21tr00:20:162
3002999129,2cyclictest0-21swapper/221:49:322
3002999129,2cyclictest0-21swapper/221:49:312
3002999128,3cyclictest8831-21cat19:30:252
3002999128,3cyclictest7584-21cat20:35:242
3002999128,3cyclictest25616-21cat20:05:252
3002999127,3cyclictest18410-21wc19:50:252
3002999120,1cyclictest0-21swapper/200:36:082
3002999120,0cyclictest0-21swapper/200:00:582
3002599129,2cyclictest4743-21cat19:25:001
3002599129,2cyclictest3826-21tr20:30:141
3002599129,2cyclictest18462-21grep21:00:121
3002599129,2cyclictest13507-21cut20:50:131
3002599127,4cyclictest976-21ssh22:13:481
3002599124,1cyclictest141rcu_preempt22:15:221
3002399129,2cyclictest17593-21tail20:55:260
3002399127,4cyclictest9087-21sed20:40:140
3003199118,2cyclictest7982-21chrt20:36:433
3003199118,2cyclictest783-21cut23:50:143
3003199118,2cyclictest5653-21cat22:35:203
3003199118,2cyclictest31369-21ssh23:29:573
3003199118,2cyclictest30781-21ssh22:30:403
3003199118,2cyclictest30687-21sh23:10:103
3003199118,2cyclictest28852-21cat22:10:253
3003199118,2cyclictest26406-21cat23:45:273
3003199118,2cyclictest26396-21cat20:10:013
3003199118,2cyclictest25736-21ssh00:05:083
3003199118,2cyclictest25577-21ssh21:50:053
3003199118,2cyclictest23045-21ssh00:22:003
3003199118,2cyclictest22105-21ssh22:25:283
3003199118,2cyclictest21870-21ssh22:06:193
3003199118,2cyclictest21142-21ssh22:45:103
3003199118,2cyclictest19933-21sed21:00:253
3003199118,2cyclictest1825-21tune2fs20:25:173
3003199118,2cyclictest14433-21tr20:50:203
3003199118,2cyclictest13351-21df22:40:143
3003199118,2cyclictest1296-21needreboot19:15:203
3003199118,2cyclictest12290-21ssh23:56:263
3003199118,2cyclictest11428-21ssh21:40:483
3003199117,3cyclictest15902-21head23:20:173
3003199110,10cyclictest3896-21expr20:30:143
3003199110,10cyclictest3347-21sh00:30:113
3002999119,1cyclictest11425-21grep19:35:262
3002999118,2cyclictest7019-21cut23:15:162
3002999118,2cyclictest5440-21expr19:25:142
3002999118,2cyclictest4034-21grep22:15:182
3002999118,2cyclictest399-21grep19:15:142
3002999118,2cyclictest31922-21sed19:10:252
3002999118,2cyclictest28474-21ssh23:46:492
3002999118,2cyclictest2664-21cat20:25:252
3002999118,2cyclictest21460-21df21:05:172
3002999118,2cyclictest2063-21grep21:55:132
3002999118,2cyclictest19728-21cat19:55:162
3002999118,2cyclictest15898-21hostname20:55:132
3002999117,3cyclictest32720-21cat20:20:272
3002999110,10cyclictest5465-21sh22:35:192
3002999110,10cyclictest26910-21ssh21:30:402
3002599118,2cyclictest6653-21tail19:25:251
3002599118,2cyclictest5977-21readlink20:35:001
3002599118,2cyclictest29057-21cut20:15:151
3002599118,2cyclictest28723-21date20:15:011
3002599118,2cyclictest12232-21uname20:45:201
3002599118,2cyclictest10229-21sed20:40:271
3002599111,6cyclictest15976-21timerandwakeup19:45:271
3002599111,5cyclictest0-21swapper/119:15:131
3002599111,2cyclictest141rcu_preempt20:07:481
3003199108,1cyclictest9368-21taskset19:33:583
3003199108,1cyclictest4939-21taskset19:25:033
3003199108,1cyclictest32084-21chrt19:11:383
3003199108,1cyclictest21502-21cat21:05:173
3003199108,1cyclictest18069-21ntpq19:50:213
3003199108,1cyclictest11913-21sh20:45:163
3003199107,2cyclictest11322-21date19:35:253
3003199106,3cyclictest915-21munin-run20:25:013
3002999108,1cyclictest29289-21expr20:15:162
3002999108,1cyclictest13981-21df_abs20:50:162
3002999108,1cyclictest139172sleep219:41:102
3002999107,2cyclictest8683-21egrep20:40:142
30029991010,0cyclictest0-21swapper/220:45:152
3002599108,1cyclictest31749-21cat19:10:241
3002599108,1cyclictest21524-21cat20:00:011
3002599108,1cyclictest1159-21cat20:25:101
3002599107,2cyclictest31292-21cut20:20:151
300259990,8cyclictest7577-21tr19:30:141
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional