You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-13 - 01:51

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Fri Jun 12, 2026 12:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
851829885,9sleep00-21swapper/007:06:480
864529468,9sleep30-21swapper/307:08:283
846829468,8sleep20-21swapper/207:06:102
844728770,13sleep10-21swapper/107:05:531
885399261,3cyclictest0-21swapper/110:05:211
8857992523,2cyclictest0-21swapper/212:25:082
8857992321,2cyclictest0-21swapper/209:46:142
885399230,22cyclictest0-21swapper/112:39:551
8848992323,0cyclictest0-21swapper/011:28:170
885799223,4cyclictest1431-21diskmemload11:05:012
885799210,1cyclictest0-21swapper/210:55:082
8853992117,3cyclictest10563-21cpuspeed_turbos09:15:151
8848992118,2cyclictest27068-21apache_accesses12:00:110
885999200,1cyclictest0-21swapper/311:15:313
885799200,20cyclictest0-21swapper/210:47:122
885799200,19cyclictest0-21swapper/212:00:332
884899206,8cyclictest7392-21idleruntime-cro08:15:000
884899203,8cyclictest0-21swapper/009:25:140
8859991914,5cyclictest0-21swapper/312:17:163
885999190,6cyclictest20941-21tr12:35:123
885799190,19cyclictest0-21swapper/211:12:502
885399190,16cyclictest0-21swapper/112:33:241
885999183,4cyclictest141rcu_preempt11:11:213
8859991812,1cyclictest0-21swapper/309:36:023
8859991810,2cyclictest29445-21cut10:05:143
8859991810,2cyclictest11860-21wc11:50:223
885799180,18cyclictest0-21swapper/211:55:282
885399183,7cyclictest0-21swapper/111:00:161
8853991815,2cyclictest13572-21cron07:20:001
884899180,18cyclictest0-21swapper/010:00:080
884899180,0cyclictest0-21swapper/009:13:440
885999172,12cyclictest19780-21apache_accesses09:40:113
8859991714,2cyclictest39-21ksoftirqd/309:13:253
885999171,11cyclictest455-21dbus-daemon11:00:003
8859991710,2cyclictest28632-21sed08:55:263
8859991710,2cyclictest10737-21grep07:10:263
8857991716,1cyclictest0-21swapper/209:31:562
8857991715,1cyclictest23589-21turbostat.cron11:04:552
8857991715,1cyclictest11399-21ssh09:35:122
885799170,17cyclictest0-21swapper/210:54:002
885799170,17cyclictest0-21swapper/209:16:072
885799170,16cyclictest0-21swapper/212:10:072
885799170,15cyclictest0-21swapper/210:38:532
885799170,0cyclictest0-21swapper/211:18:332
885799170,0cyclictest0-21swapper/207:59:072
885399174,9cyclictest225-21systemd-journal08:35:001
885399170,1cyclictest13304-21hddtemp_smartct09:55:161
884899176,5cyclictest3564-21perf12:25:000
884899170,16cyclictest0-21swapper/012:27:250
885999169,2cyclictest3748-21cat09:50:003
885999164,9cyclictest8965-21munin-node11:20:503
885999163,10cyclictest10943-21latency_hist09:35:003
8857991615,1cyclictest0-21swapper/209:54:562
885799160,16cyclictest0-21swapper/210:17:472
8853991613,3cyclictest0-21swapper/112:11:501
8853991612,3cyclictest0-21swapper/110:44:541
8853991611,1cyclictest0-21swapper/110:00:341
885399160,15cyclictest0-21swapper/112:02:471
885399160,0cyclictest0-21swapper/110:51:381
884899164,6cyclictest15246-21perf10:55:000
8848991616,0cyclictest0-21swapper/011:38:300
8848991612,2cyclictest13-21ksoftirqd/011:45:130
884899160,16cyclictest0-21swapper/012:34:390
885999155,7cyclictest325-21cstates11:05:123
885999150,3cyclictest0-21swapper/311:25:153
8857991514,1cyclictest0-21swapper/212:07:592
8857991512,2cyclictest1639-21latency_hist09:10:012
8857991511,3cyclictest20509-21head12:15:172
885799150,3cyclictest0-21swapper/210:44:152
8853991513,2cyclictest0-21swapper/107:50:001
8853991513,1cyclictest3496-21ssh12:05:141
8853991511,2cyclictest9793-21head07:10:161
8848991513,1cyclictest13-21ksoftirqd/012:15:120
884899150,7cyclictest0-21swapper/007:30:160
884899150,1cyclictest0-21swapper/010:36:060
8859991410,3cyclictest39-21ksoftirqd/311:57:283
8859991410,3cyclictest28626-21cron10:05:013
8859991410,2cyclictest10597-21grep09:15:153
885999140,4cyclictest0-21swapper/312:30:143
8857991410,3cyclictest26369-21grep11:40:142
885399147,6cyclictest28452-21grep09:25:221
8853991413,1cyclictest0-21swapper/108:40:141
8853991413,1cyclictest0-21swapper/108:40:141
8853991410,3cyclictest27512-21cron12:20:001
885399140,2cyclictest0-21swapper/111:19:241
885399140,14cyclictest0-21swapper/110:25:051
884899149,4cyclictest94-21kswapd011:43:260
884899145,2cyclictest7947-21munin-run11:10:000
8848991412,1cyclictest13-21ksoftirqd/009:48:350
8848991411,2cyclictest1431-21diskmemload12:05:400
8848991410,3cyclictest13843-21grep09:55:220
8848991410,2cyclictest11569-21strings09:15:210
884899140,2cyclictest0-21swapper/011:50:280
885999136,3cyclictest225-21systemd-journal10:40:013
8859991312,1cyclictest0-21swapper/310:18:133
8859991310,2cyclictest6609-21wc08:10:243
8859991310,2cyclictest25480-21cron07:45:013
8859991310,2cyclictest25011-21tail11:00:183
8859991310,2cyclictest24334-21cat08:50:003
885999130,3cyclictest225-21systemd-journal10:29:433
885799139,3cyclictest30466-21grep09:00:212
885799139,3cyclictest28482-21cat08:55:262
885799139,3cyclictest24056-21sh08:50:002
8857991310,2cyclictest28894-21sed07:50:172
8857991310,2cyclictest28117-21cat09:25:192
8857991310,2cyclictest22012-21grep10:00:192
8857991310,2cyclictest15086-21tr08:30:152
885399139,3cyclictest825-21tail11:05:141
885399134,8cyclictest21488-21ssh11:36:491
8853991312,1cyclictest38150irq/131-snd_hda10:20:171
8853991311,2cyclictest55650irq/127-enp2s0-10:31:591
8853991310,2cyclictest931-21grep10:45:261
8853991310,2cyclictest23118-21tr07:40:001
8853991310,2cyclictest15413-21grep10:15:271
8853991310,2cyclictest14616-21sed07:20:171
884899131,6cyclictest27880-21awk07:50:000
8848991311,1cyclictest15767-21apache_volume10:55:130
8848991310,2cyclictest29420-21cut10:05:140
8848991310,2cyclictest27016-21tr08:55:140
8848991310,2cyclictest14575-21head07:20:170
8848991310,2cyclictest13-21ksoftirqd/008:40:000
884899130,3cyclictest0-21swapper/011:59:430
884899130,1cyclictest141rcu_preempt09:22:190
884899130,13cyclictest0-21swapper/012:10:560
884899130,12cyclictest5595-21sh09:50:240
885999129,3cyclictest29883-21apache211:31:223
885999129,2cyclictest6472-21head10:10:203
885999129,2cyclictest5984-21grep12:25:253
885999129,2cyclictest24919-21grep10:40:263
885999129,2cyclictest24608-21grep08:50:143
885999129,2cyclictest20679-21head09:20:283
885999129,2cyclictest15555-21sed07:20:283
885999129,2cyclictest12430-21apt-get09:55:133
885999129,2cyclictest10849-21cut08:20:173
885999128,3cyclictest8159-21grep10:50:163
885999128,3cyclictest32290-21grep09:05:163
885999128,3cyclictest32219-21sed07:55:243
885999121,11cyclictest0-21swapper/312:05:583
885999120,11cyclictest19198-21ssh11:35:223
885999120,11cyclictest0-21swapper/310:33:103
885799129,2cyclictest8425-21grep08:15:172
885799129,2cyclictest18495-21head11:35:172
885799129,2cyclictest17465-21grep08:35:142
885799129,2cyclictest13119-21sed07:15:272
885799128,3cyclictest5269-21ssh10:10:132
885799120,4cyclictest0-21swapper/212:34:272
885799120,1cyclictest0-21swapper/211:51:052
885399129,2cyclictest4109-21uname09:30:201
885399129,2cyclictest4059-21basename09:50:131
885399129,2cyclictest30657-21cut07:55:121
885399129,2cyclictest29450-21head12:20:221
885399129,2cyclictest29032-21ls09:00:001
885399129,2cyclictest26018-21sed11:20:171
885399129,2cyclictest16965-21head07:25:201
885399128,3cyclictest11678-21tr09:35:131
885399128,1cyclictest30896-21ssh09:45:591
885399124,4cyclictest225-21systemd-journal07:55:001
885399124,2cyclictest225-21systemd-journal08:05:001
885399122,9cyclictest38150irq/131-snd_hda09:00:131
8853991212,0cyclictest0-21swapper/110:56:491
8853991211,0cyclictest0-21swapper/111:55:041
885399120,12cyclictest1431-21diskmemload10:11:251
885399120,12cyclictest0-21swapper/109:43:401
885399120,11cyclictest17401-21date11:35:001
885399120,10cyclictest0-21swapper/110:38:301
884899129,2cyclictest9795-21cat08:20:010
884899129,2cyclictest977-21tail09:05:260
884899129,2cyclictest6996-21grep10:10:240
884899129,2cyclictest6335-21grep10:30:120
884899129,2cyclictest31735-21tr10:45:140
884899129,2cyclictest15433-21cut08:30:170
884899128,3cyclictest31079-21sed09:00:280
884899126,3cyclictest13-21ksoftirqd/010:42:140
884899123,4cyclictest225-21systemd-journal08:40:000
884899123,4cyclictest225-21systemd-journal08:40:000
8848991211,1cyclictest0-21swapper/011:11:200
884899121,11cyclictest0-21swapper/010:17:060
884899120,1cyclictest0-21swapper/011:33:250
885999118,2cyclictest8404-21grep08:15:173
885999118,2cyclictest4636-21ssh09:50:173
885999118,2cyclictest4008-21sed08:05:213
885999118,2cyclictest29499-21cut12:20:223
885999118,2cyclictest27821-21cat11:40:253
885999118,2cyclictest26972-21ssh12:00:083
885999118,2cyclictest26833-21sh09:25:053
885999118,2cyclictest2542-21expr11:45:143
885999118,2cyclictest23002-21wc10:20:183
885999118,2cyclictest2210-21rm10:46:223
885999118,2cyclictest17732-21grep08:35:163
885999118,2cyclictest1678-21grep08:00:213
885999118,2cyclictest15588-21ssh12:12:113
885799118,2cyclictest8974-21cat11:30:002
885799118,2cyclictest4469-21ssh09:10:502
885799118,2cyclictest28946-21ssh10:05:052
885799118,2cyclictest2799-21grep11:45:162
885799118,2cyclictest27785-21ssh12:20:022
885799118,2cyclictest26165-21grep08:50:272
885799118,2cyclictest24717-21ssh10:21:072
885799118,2cyclictest23667-21tr07:40:172
885799118,2cyclictest23506-21ssh09:41:472
885799118,2cyclictest21642-21ssh09:21:112
885799118,2cyclictest21205-21ssh12:35:132
885799118,2cyclictest20293-21head08:40:172
885799118,2cyclictest20293-21head08:40:172
885799118,2cyclictest19533-21irqstats07:30:222
885799118,2cyclictest13052-21tune2fs08:25:172
885799118,2cyclictest1285-21sed08:00:162
885799118,2cyclictest12685-21tr09:55:142
885799118,2cyclictest1180-21ssh10:26:302
885799118,2cyclictest11572-21ssh11:30:352
885799118,2cyclictest10651-21taskset08:20:162
885799117,2cyclictest16913-21grep07:25:202
885799110,1cyclictest0-21swapper/211:23:212
885799110,10cyclictest6980-21sed08:10:282
885799110,10cyclictest0-21swapper/210:30:052
885399119,1cyclictest212182sleep109:20:521
885399118,2cyclictest8172-21ssh09:13:441
885399118,2cyclictest7725-21grep08:15:131
885399118,2cyclictest6000-21ssh11:46:571
885399118,2cyclictest5260-21basename08:10:131
885399118,2cyclictest4303-21tr12:25:141
885399118,2cyclictest28599-21ssh11:40:501
885399118,2cyclictest22728-21grep08:45:181
885399118,2cyclictest14108-21grep08:25:261
885399118,2cyclictest1395-21cut11:25:141
885399118,2cyclictest13167-21ssh11:12:331
885399118,2cyclictest12615-21ssh11:50:351
885399117,2cyclictest24678-21tr08:50:141
884899118,2cyclictest6497-21ssh09:31:340
884899118,2cyclictest31905-21mailstats07:55:200
884899118,2cyclictest31643-21tr10:25:200
884899118,2cyclictest28667-21df_abs07:50:160
884899118,2cyclictest28470-21ssh11:21:280
884899118,2cyclictest27174-21ssh11:01:160
884899118,2cyclictest22649-21grep12:35:240
884899118,2cyclictest20297-21expr09:40:140
884899118,2cyclictest17860-21grep07:25:300
884899118,2cyclictest13526-21ssh09:35:320
884899118,2cyclictest13-21ksoftirqd/008:05:260
884899118,2cyclictest10404-21wc07:10:220
884899118,2cyclictest10207-21egrep08:20:140
884899117,3cyclictest29703-21ls10:25:000
885999108,1cyclictest26093-21expr07:45:153
885999108,1cyclictest20960-21basename07:35:133
885999108,1cyclictest12100-21grep07:15:173
885999107,2cyclictest29943-21df09:00:173
885999107,2cyclictest18608-21apache_accesses07:30:153
885999107,2cyclictest15274-21cat08:30:153
885999107,2cyclictest12574-21grep08:25:143
885999100,9cyclictest17727-21cat07:25:293
885799108,1cyclictest9536-21cat07:10:152
885799108,1cyclictest3518-21cut08:05:172
885799108,1cyclictest138202sleep207:20:052
885799107,2cyclictest26671-21irqstats07:45:192
885799107,2cyclictest21678-21tune2fs07:35:192
885399108,1cyclictest24508-21ntp_kernel_pll_07:40:231
885399107,2cyclictest18723-21aten2_r7power_v07:30:161
885399107,2cyclictest10690-21sort08:20:171
884899108,1cyclictest22852sleep008:00:330
884899108,1cyclictest143152sleep008:27:140
884899107,2cyclictest24830-21cut08:50:150
884899107,2cyclictest24154-21sed07:40:190
884899107,2cyclictest23063-21uname08:45:220
884899107,2cyclictest22641-21grep07:35:290
884899107,2cyclictest16692-21ssh11:15:080
884899107,2cyclictest12113-21grep07:15:170
88599997,1cyclictest21448-21chrt08:42:073
88599997,1cyclictest21448-21chrt08:42:073
88599996,2cyclictest28725-21sort07:50:183
88539997,1cyclictest3199-21grep08:05:151
88539997,1cyclictest31711-21acpi09:05:111
88539996,2cyclictest18144-21wc08:35:191
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional