You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-19 - 08:12

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Sat Sep 19, 2026 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1460821360,0sleep30-21swapper/323:07:423
1899729679,10sleep30-21swapper/319:07:273
1885329279,9sleep00-21swapper/019:05:360
1885028875,9sleep20-21swapper/219:05:332
1906028467,13sleep10-21swapper/119:08:181
238122670,0sleep30-21swapper/323:32:423
41652650,0sleep10-21swapper/121:43:521
19283993429,4cyclictest1547-21ssh23:39:063
19275993228,1cyclictest13-21ksoftirqd/000:05:570
1928399310,19cyclictest16783-21kworker/3:019:52:463
19275993112,5cyclictest13-21ksoftirqd/023:36:120
1928399272,24cyclictest0-21swapper/323:17:463
1927799262,23cyclictest13621-21tail23:45:261
1927599250,21cyclictest25529-21sh23:34:030
1928199240,24cyclictest0-21swapper/200:09:392
1928199230,23cyclictest0-21swapper/200:18:522
1927799230,2cyclictest0-21swapper/100:29:281
1927799230,20cyclictest0-21swapper/121:10:491
19275992321,1cyclictest0-21swapper/000:33:200
19275992319,3cyclictest4488-21ssh00:00:130
19275992319,3cyclictest4488-21ssh00:00:120
1927599230,21cyclictest0-21swapper/023:03:070
1928399222,19cyclictest16783-21kworker/3:019:56:413
19283992214,6cyclictest29588-21/usr/sbin/munin00:15:153
1927799210,19cyclictest0-21swapper/100:32:041
52432200,0sleep00-21swapper/022:42:340
1928199203,15cyclictest0-21swapper/223:35:502
1928199200,1cyclictest0-21swapper/200:21:262
1927799200,18cyclictest16913-21awk20:10:221
1927599205,5cyclictest13-21ksoftirqd/022:06:070
1927599200,4cyclictest0-21swapper/022:31:450
1927599200,1cyclictest0-21swapper/000:29:370
1927799191,1cyclictest8117-21expr21:00:161
1927799190,1cyclictest11906-21diskmemload22:13:571
1927599195,7cyclictest17558-21ssh23:10:000
19275991918,1cyclictest161rcuc/000:15:140
19275991915,3cyclictest13-21ksoftirqd/022:25:070
19275991915,3cyclictest13-21ksoftirqd/022:25:060
273262180,0sleep30-21swapper/322:16:523
1928199182,15cyclictest10415-21runrttasks21:28:062
1928199180,18cyclictest0-21swapper/223:40:142
19275991812,1cyclictest13-21ksoftirqd/023:58:540
1927599180,7cyclictest0-21swapper/022:02:530
1927599180,4cyclictest0-21swapper/022:36:080
1928399171,0cyclictest0-21swapper/322:30:553
1928199170,0cyclictest0-21swapper/222:40:052
19277991716,1cyclictest0-21swapper/119:30:111
1927799171,2cyclictest0-21swapper/122:25:301
1927799171,2cyclictest0-21swapper/122:25:291
1927799170,3cyclictest0-21swapper/121:58:431
1927599171,9cyclictest0-21swapper/022:45:120
19275991715,2cyclictest13-21ksoftirqd/022:57:240
19275991715,2cyclictest13-21ksoftirqd/021:35:200
1927599171,2cyclictest0-21swapper/023:54:230
1927599171,1cyclictest141rcu_preempt22:50:270
1927599170,4cyclictest13-21ksoftirqd/023:22:220
1927599170,17cyclictest0-21swapper/021:45:130
164792170,0sleep20-21swapper/221:31:112
1928399164,7cyclictest21883-21apt-get21:35:153
19283991612,2cyclictest0-21swapper/321:11:473
1928399161,15cyclictest0-21swapper/300:26:343
1928199164,1cyclictest141rcu_preempt00:26:112
1928199162,2cyclictest141rcu_preempt23:56:112
19281991615,1cyclictest32-21ksoftirqd/222:33:252
19281991614,2cyclictest55550irq/126-enp2s0-00:34:402
1928199160,2cyclictest0-21swapper/221:39:142
1928199160,16cyclictest0-21swapper/222:02:292
1928199160,12cyclictest0-21swapper/219:55:132
1927799161,1cyclictest3231-21ssh23:20:191
1927799160,1cyclictest0-21swapper/121:27:471
1927799160,15cyclictest0-21swapper/122:53:501
1927599165,5cyclictest13-21ksoftirqd/021:15:280
1927599163,7cyclictest11682-21apache_volume23:45:120
19275991615,1cyclictest0-21swapper/022:10:170
19275991612,1cyclictest13-21ksoftirqd/021:57:110
1927599160,5cyclictest0-21swapper/022:24:110
1927599160,2cyclictest0-21swapper/021:31:560
1927599160,2cyclictest0-21swapper/000:39:040
19283991512,3cyclictest39-21ksoftirqd/322:57:403
19283991511,3cyclictest25147-21cut22:35:173
1928399150,4cyclictest0-21swapper/321:54:393
1928399150,0cyclictest0-21swapper/322:22:213
19281991510,4cyclictest0-21swapper/221:14:252
19277991510,4cyclictest0-21swapper/123:13:181
1927599154,1cyclictest7972-21taskset19:50:590
19275991513,1cyclictest11906-21diskmemload21:25:120
19275991512,3cyclictest161rcuc/021:54:260
1928399144,2cyclictest0-21swapper/321:30:283
19283991411,2cyclictest29076-21cut23:55:173
19283991410,3cyclictest9852-21grep21:05:143
19283991410,2cyclictest21161-21grep19:10:263
1928399140,13cyclictest0-21swapper/323:41:493
1928199149,2cyclictest32-21ksoftirqd/222:05:222
1928199148,0cyclictest141rcu_preempt00:04:292
1928199148,0cyclictest141rcu_preempt00:04:282
19281991412,2cyclictest32-21ksoftirqd/223:00:562
19281991411,2cyclictest11413-21grep23:25:162
1928199140,0cyclictest0-21swapper/221:23:492
19277991412,1cyclictest0-21swapper/123:25:141
19277991411,3cyclictest38150irq/131-snd_hda22:24:101
1927799140,2cyclictest0-21swapper/100:24:501
1927799140,1cyclictest0-21swapper/123:50:581
1927799140,1cyclictest0-21swapper/123:40:361
1927799140,0cyclictest0-21swapper/122:34:471
1927599148,2cyclictest31312-21cron22:20:000
1927599146,3cyclictest1-21systemd20:28:320
1927599144,5cyclictest225-21systemd-journal21:21:230
1927599143,7cyclictest13-21ksoftirqd/021:44:590
1927599143,3cyclictest9397-21chrt21:03:360
19275991410,3cyclictest9980-21wc19:55:240
19275991410,3cyclictest5378-21sed23:40:260
19275991410,3cyclictest31948-21grep20:40:250
239822130,2sleep125-21ksoftirqd/100:10:581
1928399139,3cyclictest30737-21grep19:30:253
1928399139,3cyclictest12001-21grep23:45:143
1928399139,3cyclictest10737-21grep23:25:143
1928399134,5cyclictest7407-21apache_volume21:00:113
19283991313,0cyclictest0-21swapper/300:34:273
19283991312,1cyclictest0-21swapper/300:05:143
19283991310,2cyclictest5701-21tail00:00:193
19283991310,2cyclictest5701-21tail00:00:183
19283991310,2cyclictest5304-21grep20:55:153
19283991310,2cyclictest4749-21idleruntime-cro20:55:003
19283991310,2cyclictest2218-21sed19:40:153
19283991310,2cyclictest17435-21cut22:50:143
19283991310,2cyclictest16962-21wc22:10:273
19283991310,2cyclictest1476-21cut20:45:223
1928399130,2cyclictest0-21swapper/323:00:193
1928399130,0cyclictest0-21swapper/300:36:563
1928199133,8cyclictest15608-21needreboot21:50:182
19281991310,2cyclictest31826-21cat20:40:252
19281991310,2cyclictest31346-21tune2fs00:35:182
19281991310,2cyclictest29986-21sleep20:40:012
19281991310,2cyclictest19764-21head23:30:172
19281991310,2cyclictest16059-21grep22:10:182
1928199130,1cyclictest0-21swapper/223:20:152
1928199130,13cyclictest0-21swapper/222:26:392
1928199130,13cyclictest0-21swapper/222:26:382
1927799139,3cyclictest9254-21sh23:04:591
1927799138,3cyclictest25520-21grep19:20:211
19277991311,2cyclictest38150irq/131-snd_hda22:08:131
19277991310,2cyclictest27712-21grep23:15:201
19277991310,2cyclictest15245-21ls21:30:251
19277991310,2cyclictest1463-21ls20:45:221
1927599136,7cyclictest13-21ksoftirqd/021:14:310
1927599131,6cyclictest4754-21munin-run20:55:010
19275991310,2cyclictest10566-21sed21:05:160
1928399129,2cyclictest4248-21grep23:20:273
1928399129,2cyclictest32071-21grep21:40:263
1928399129,2cyclictest31975-21head20:40:263
1928399129,2cyclictest28354-21grep20:35:153
1928399129,2cyclictest15008-21grep20:05:263
1928399129,2cyclictest10740-21ssh22:06:513
1928399128,3cyclictest9043-21tr22:25:203
1928399128,3cyclictest9043-21tr22:25:193
1928399128,3cyclictest904-21cut22:40:173
1928399120,12cyclictest0-21swapper/300:14:193
1928199129,2cyclictest9012-21grep21:00:252
1928199129,2cyclictest28275-21grep19:25:272
1928199129,2cyclictest25555-21cat22:35:192
1928199129,2cyclictest19475-21tr20:15:192
1928199129,2cyclictest13652-21grep20:05:142
1928199129,2cyclictest12728-21ssh21:49:072
1928199128,3cyclictest32019-21sed21:40:252
19281991210,1cyclictest18064-21cat20:15:012
19281991210,1cyclictest1154-21latency_hist19:40:002
19281991210,1cyclictest0-21swapper/219:50:112
1928199120,12cyclictest0-21swapper/221:59:522
1928199120,11cyclictest18930-21sed23:10:172
1928199120,11cyclictest0-21swapper/223:47:322
1927799129,2cyclictest8402-21cat19:55:001
1927799129,2cyclictest5948-21apt00:39:591
1927799129,2cyclictest500-21grep22:00:271
1927799129,2cyclictest22220-21wc21:15:201
1927799129,2cyclictest22177-21grep19:15:141
1927799129,2cyclictest20856-21chrt19:10:221
1927799129,2cyclictest14722-21sed00:05:271
1927799128,3cyclictest31552-21tr19:35:161
1927799127,4cyclictest29645-21sh00:15:141
1927799124,4cyclictest25-21ksoftirqd/123:30:171
19277991210,2cyclictest38150irq/131-snd_hda21:39:251
1927799120,2cyclictest0-21swapper/121:47:261
1927799120,1cyclictest0-21swapper/123:35:161
1927599129,2cyclictest5855-21grep00:20:140
1927599129,2cyclictest23122-21head19:15:210
1927599129,2cyclictest17419-21grep20:10:270
1927599129,2cyclictest15508-21cat20:09:590
1927599128,3cyclictest6677-21grep20:55:260
1927599128,3cyclictest28263-21sed23:15:240
1927599126,5cyclictest0-21swapper/023:26:340
1927599121,3cyclictest225-21systemd-journal19:10:580
19275991211,0cyclictest0-21swapper/000:12:200
19275991210,1cyclictest11906-21diskmemload23:10:040
1928399118,2cyclictest9032-21ssh21:46:103
1928399118,2cyclictest6947-21ssh21:25:233
1928399118,2cyclictest6893-21grep00:20:203
1928399118,2cyclictest27404-21tail19:25:173
1928399118,2cyclictest26019-21ssh21:56:303
1928399118,2cyclictest25987-21head19:20:253
1928399118,2cyclictest25450-21chrt20:30:063
1928399118,2cyclictest21539-21grep21:15:153
1928399118,2cyclictest19781-21ssh23:50:023
1928399118,2cyclictest11493-21ssh22:45:563
1928399110,10cyclictest29087-21ssh21:20:123
1928399110,10cyclictest2090-21ssh22:01:333
1928199119,2cyclictest291irq_work/222:23:462
1928199119,1cyclictest26392-21cron19:25:002
1928199119,1cyclictest11484-21grep21:05:262
1928199118,2cyclictest27415-21head22:55:262
1928199118,2cyclictest25238-21sleep20:30:002
1928199118,2cyclictest24537-21rm23:52:242
1928199118,2cyclictest24065-21cat19:19:592
1928199118,2cyclictest2368-21cron20:50:012
1928199118,2cyclictest23266-21ssh22:15:022
1928199118,2cyclictest18612-21cut22:50:222
1928199118,2cyclictest13284-21cut20:05:012
1928199114,6cyclictest32-21ksoftirqd/221:15:122
1928199114,0cyclictest141rcu_preempt22:45:292
1927799118,2cyclictest9581-21ssh23:05:061
1927799118,2cyclictest9469-21ssh22:45:151
1927799118,2cyclictest6410-21ssh00:00:261
1927799118,2cyclictest6410-21ssh00:00:251
1927799118,2cyclictest31497-21sh21:20:461
1927799118,2cyclictest30615-21expr20:40:131
1927799118,2cyclictest28802-21expr23:55:151
1927799118,2cyclictest28704-21ssh22:56:241
1927799118,2cyclictest25215-21cron20:30:001
1927799118,2cyclictest24681-21grep22:35:141
1927799118,2cyclictest19825-21cat20:15:241
1927799118,2cyclictest15453-21ls21:50:171
1927799117,3cyclictest6702-21sed20:55:271
1927799117,3cyclictest22956-21cat20:25:011
1927799116,3cyclictest3581-21sh22:41:151
1927599118,2cyclictest5038-21head19:45:210
1927599118,2cyclictest29673-21chrt20:36:200
1927599118,2cyclictest19049-21tail20:15:150
1927599118,2cyclictest1157-21sendmail19:40:000
19275991110,1cyclictest13-21ksoftirqd/020:21:010
1928399108,1cyclictest24330-21grep20:25:193
1928399108,1cyclictest22498-21cut19:15:153
1928399107,2cyclictest3942-21cut19:45:143
1928399107,2cyclictest20449-21cron20:20:003
1928399107,2cyclictest18214-21cut23:10:143
1928399107,2cyclictest17085-21tr20:10:233
1928399107,2cyclictest12258-21cut20:00:213
1928399105,2cyclictest225-21systemd-journal19:40:003
1928399100,9cyclictest213332sleep320:20:163
1928199108,2cyclictest3310-21seq19:42:012
1928199108,1cyclictest9544-21ssh23:05:042
1928199108,1cyclictest28565-21taskset23:15:372
1928199107,2cyclictest4050-21tr19:45:142
1928199107,2cyclictest26103-21grep20:30:162
1928199107,2cyclictest22085-21sh00:10:152
1928199107,2cyclictest21230-21sed19:10:262
1927799108,1cyclictest56902sleep119:47:081
1927799108,1cyclictest26570-21cat20:30:191
1927799108,1cyclictest23323-21ssh22:15:041
1927799108,1cyclictest116492sleep121:07:011
1927799107,2cyclictest26988-21cat19:25:161
1927799107,2cyclictest2605-21chrt20:50:061
1927799107,2cyclictest15224-21chrt20:06:511
1927799107,2cyclictest12023-21cat20:00:191
1927799103,7cyclictest19792-21munin-node20:36:551
1927599108,2cyclictest26842-21mailstats20:30:210
1927599108,1cyclictest31117-21taskset19:34:460
1927599107,2cyclictest24966-21cut19:20:150
1927599103,3cyclictest225-21systemd-journal20:50:010
1927599100,9cyclictest27617-21cut19:25:200
192819998,1cyclictest225652sleep220:21:272
192819997,2cyclictest29038-21apt19:30:122
192819997,1cyclictest6925-21taskset20:57:352
192819997,1cyclictest2600-21taskset20:50:022
192779997,1cyclictest8540-21cat19:55:111
192779996,2cyclictest1747-21cat19:40:131
192759996,2cyclictest1580-21aten2_r7power_v19:40:130
192759996,2cyclictest11197-21cut20:00:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional