You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-04 - 22:32

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Sat Jul 04, 2026 12:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2697228673,9sleep00-21swapper/007:08:070
2697228673,9sleep00-21swapper/007:08:060
2693628568,13sleep10-21swapper/107:07:401
2693628568,13sleep10-21swapper/107:07:391
2708628370,9sleep20-21swapper/207:09:382
2708628370,9sleep20-21swapper/207:09:382
2499028370,9sleep30-21swapper/307:05:093
2499028370,9sleep30-21swapper/307:05:083
201802710,0sleep00-21swapper/011:23:140
88632700,0sleep20-21swapper/210:38:332
27214992824,3cyclictest94-21kswapd009:35:123
27214992626,0cyclictest0-21swapper/311:23:173
27214992319,3cyclictest0-21swapper/310:20:123
253572230,0sleep10-21swapper/109:31:071
2721499225,16cyclictest0-21swapper/311:18:413
2721499220,20cyclictest0-21swapper/312:21:203
2721299220,22cyclictest0-21swapper/211:47:362
2721299220,22cyclictest0-21swapper/211:47:352
2721299220,22cyclictest0-21swapper/210:54:592
27209992213,8cyclictest25-21ksoftirqd/111:34:531
27214992120,1cyclictest20479-21diskmemload09:47:133
27214992119,1cyclictest25270-21apt-get10:10:133
27214992111,6cyclictest20479-21diskmemload10:48:583
27212992120,1cyclictest0-21swapper/208:55:132
2721299210,2cyclictest0-21swapper/209:43:172
2721299210,21cyclictest0-21swapper/211:20:212
2720999217,7cyclictest537-21snmpd10:32:431
2720999210,20cyclictest0-21swapper/109:57:271
2721499209,8cyclictest24182-21missed_timers11:25:203
2721299200,20cyclictest0-21swapper/210:27:492
2720999200,19cyclictest0-21swapper/111:05:461
2721499199,7cyclictest20479-21diskmemload09:20:093
2721499192,16cyclictest0-21swapper/311:57:263
27214991917,1cyclictest20479-21diskmemload10:32:363
27214991915,3cyclictest14762-21ssh09:25:173
27214991912,7cyclictest20479-21diskmemload11:00:493
2721299190,3cyclictest0-21swapper/212:21:072
2721299190,2cyclictest0-21swapper/210:43:142
2721299190,19cyclictest0-21swapper/212:39:232
2721299190,18cyclictest0-21swapper/211:05:072
12752190,0sleep00-21swapper/012:09:550
2721499188,8cyclictest0-21swapper/311:08:213
2721499187,3cyclictest0-21swapper/311:30:573
2721499185,12cyclictest0-21swapper/310:27:033
27214991817,1cyclictest20479-21diskmemload12:07:373
27214991810,7cyclictest4560-21awk07:30:003
2721299180,18cyclictest0-21swapper/211:16:262
2721299180,18cyclictest0-21swapper/209:32:562
2720999186,11cyclictest25-21ksoftirqd/112:25:031
2720799186,6cyclictest9289-21munin-run07:40:010
2720799185,6cyclictest8320-21munin-run08:45:010
2720799180,15cyclictest0-21swapper/010:42:560
2721499178,8cyclictest0-21swapper/310:01:073
2721499172,12cyclictest0-21swapper/309:18:383
27214991711,4cyclictest20479-21diskmemload11:45:523
27214991711,4cyclictest20479-21diskmemload11:45:513
27214991711,4cyclictest20479-21diskmemload09:11:203
27214991711,2cyclictest0-21swapper/309:40:053
27212991717,0cyclictest0-21swapper/211:54:312
27212991716,1cyclictest32-21ksoftirqd/209:48:452
27212991715,1cyclictest3291-21pmu-power12:10:212
2721299170,17cyclictest0-21swapper/212:15:452
2721299170,0cyclictest0-21swapper/209:24:002
2720799175,9cyclictest13921-21/usr/sbin/munin09:25:200
2720799172,2cyclictest0-21swapper/011:33:300
27207991711,3cyclictest0-21swapper/010:22:160
2720799170,16cyclictest0-21swapper/012:35:120
302622160,0sleep30-21swapper/309:53:543
2721499169,6cyclictest20479-21diskmemload11:51:553
2721499164,2cyclictest0-21swapper/309:59:013
27214991614,1cyclictest20479-21diskmemload12:28:223
27214991614,1cyclictest20479-21diskmemload12:19:413
27214991614,1cyclictest20479-21diskmemload09:34:073
27214991611,2cyclictest0-21swapper/310:37:513
2721299161,11cyclictest0-21swapper/211:03:472
2721299160,16cyclictest0-21swapper/209:37:302
2721299160,15cyclictest0-21swapper/211:57:442
2721299160,15cyclictest0-21swapper/210:08:592
2720999161,4cyclictest141rcu_preempt09:48:201
27209991614,1cyclictest0-21swapper/110:35:201
27209991610,6cyclictest9448-21apache211:12:271
2720999160,14cyclictest0-21swapper/109:37:281
2720999160,12cyclictest0-21swapper/112:13:111
2720799160,1cyclictest0-21swapper/009:20:390
221082160,0chrt0-21swapper/212:02:282
27214991513,1cyclictest20479-21diskmemload11:11:123
2721299151,5cyclictest141rcu_preempt10:30:142
2721299150,15cyclictest0-21swapper/211:12:392
27209991513,1cyclictest0-21swapper/108:35:241
2720999150,14cyclictest9146-21grep10:00:161
27207991511,3cyclictest26921-21wc12:25:000
27207991510,5cyclictest1370-21apache212:01:150
2720799150,15cyclictest0-21swapper/009:33:240
2721499148,4cyclictest20479-21diskmemload10:07:113
2721499144,8cyclictest20479-21diskmemload11:39:553
2721499142,4cyclictest0-21swapper/312:02:003
2721499142,11cyclictest13786-21grep12:35:253
27214991412,1cyclictest20479-21diskmemload12:12:283
27214991412,1cyclictest0-21swapper/310:15:163
2721299149,3cyclictest29494-21sort08:20:172
2721299140,14cyclictest0-21swapper/210:56:532
2721299140,14cyclictest0-21swapper/209:59:342
2720999144,8cyclictest11995-21/usr/sbin/munin12:35:131
2720999141,12cyclictest0-21swapper/112:15:161
27209991410,3cyclictest4778-21awk12:30:221
27209991410,2cyclictest9332-21tr11:55:171
27209991410,2cyclictest20761-21grep08:00:251
2720999140,2cyclictest0-21swapper/110:08:541
2720799145,4cyclictest455-21dbus-daemon09:59:590
2720799144,6cyclictest1438-21apt-get08:30:130
27207991410,2cyclictest8165-21cut11:35:170
2720799140,2cyclictest0-21swapper/011:25:040
2720799140,11cyclictest0-21swapper/009:43:000
2721499139,3cyclictest7458-21wc08:40:223
2721499138,4cyclictest0-21swapper/308:15:203
2721499136,5cyclictest9282-21latency_hist07:40:003
2721499135,8cyclictest0-21swapper/310:50:563
27214991311,1cyclictest20479-21diskmemload10:41:233
27214991310,1cyclictest20479-21diskmemload12:32:403
2721499130,1cyclictest0-21swapper/309:00:143
2721299135,2cyclictest225-21systemd-journal07:22:362
27212991311,1cyclictest32-21ksoftirqd/211:26:442
27212991310,2cyclictest32-21ksoftirqd/211:40:222
27212991310,2cyclictest29042-21wc12:25:262
27212991310,2cyclictest2106-21grep08:30:172
27212991310,2cyclictest10366-21grep07:40:172
2721299130,13cyclictest0-21swapper/211:33:182
2721299130,12cyclictest3677-21tr12:30:152
2720999138,5cyclictest9452-21apache212:06:491
2720999138,4cyclictest455-21dbus-daemon11:20:001
2720999138,4cyclictest18782-21taskset07:59:071
2720999131,1cyclictest28711-21sed08:19:591
27209991310,2cyclictest9864-21tr07:40:141
27209991310,2cyclictest8310-21turbostat08:45:011
27209991310,2cyclictest29686-21tr09:15:141
27209991310,2cyclictest1896-21cron07:25:011
27209991310,2cyclictest17689-21wc11:40:251
2720999130,0cyclictest0-21swapper/109:10:521
2720799139,3cyclictest32263-21sed08:25:160
2720799139,3cyclictest31233-21grep09:35:130
2720799134,8cyclictest29566-21perf07:15:010
2720799131,11cyclictest30533-21grep08:20:280
27207991310,2cyclictest2999-21sed12:10:200
27207991310,2cyclictest29588-21cat10:50:190
27207991310,2cyclictest25558-21tail08:10:260
27207991310,2cyclictest23741-21wc08:09:590
27207991310,2cyclictest22876-21sed09:10:250
27207991310,2cyclictest18331-21cron12:20:000
27207991310,2cyclictest15144-21grep07:50:150
27207991310,2cyclictest14218-21grep08:55:170
2720799130,11cyclictest17990-21grep11:40:260
2721499129,2cyclictest5175-21chrt10:55:163
2721499129,2cyclictest29082-21grep07:10:273
2721499129,2cyclictest18452-21grep07:55:283
2721499129,2cyclictest15088-21grep07:50:153
2721499128,3cyclictest21256-21sh08:04:593
27214991210,2cyclictest0-21swapper/311:41:403
2721299129,2cyclictest26031-21ssh12:05:142
2721299129,2cyclictest16675-21wc07:54:592
2721299129,2cyclictest14059-21taskset09:25:142
2721299129,1cyclictest1974-21diskstats10:15:172
2721299128,3cyclictest9094-21tail10:00:152
27212991212,0cyclictest0-21swapper/210:14:472
2720999129,2cyclictest8699-21grep11:35:211
2720999129,2cyclictest8412-21grep07:35:201
2720999129,2cyclictest732-21grep08:25:251
2720999129,2cyclictest6262-21grep09:20:161
2720999129,2cyclictest5942-21tr07:30:201
2720999129,2cyclictest3654-21sort07:25:251
2720999129,2cyclictest30122-21tr08:20:241
2720999129,2cyclictest28105-21cut07:10:181
2720999129,2cyclictest25565-21expr10:10:131
2720999129,2cyclictest24465-21ssh09:50:141
2720999129,2cyclictest1514-21grep10:15:151
2720999129,2cyclictest15094-21grep07:50:151
2720999129,2cyclictest14854-21tr11:20:151
2720999128,2cyclictest19876-21wc09:05:261
2720999128,2cyclictest13204-21grep11:00:151
2720999121,10cyclictest20479-21diskmemload09:27:011
2720799129,3cyclictest9452-21apache210:06:280
2720799129,2cyclictest4977-21rm10:55:150
2720799129,2cyclictest3750-21head10:35:180
2720799129,2cyclictest3299-21tr07:25:200
2720799129,2cyclictest28625-21systemctl08:19:590
2720799129,2cyclictest19638-21df08:00:160
2720799129,2cyclictest19119-21ssh10:25:200
2720799129,2cyclictest14297-21wc07:50:000
2720799128,3cyclictest24188-21sh09:50:130
2720799126,5cyclictest9448-21apache210:15:130
87582110,1sleep10-21swapper/108:45:121
2721499119,1cyclictest67312sleep307:32:503
2721499119,1cyclictest15238-21grep08:55:263
2721499118,2cyclictest32759-21df07:20:153
2721499118,2cyclictest32335-21cat08:25:183
2721499118,2cyclictest3103-21grep08:30:253
2721499118,2cyclictest29672-21head08:20:183
2721499118,2cyclictest25575-21sed08:10:253
2721499118,2cyclictest12738-21cat08:50:263
2721499118,2cyclictest10822-21/usr/sbin/munin07:40:223
2721499116,4cyclictest31953-21turbostat07:20:013
2721499114,2cyclictest0-21swapper/307:45:143
2721499110,1cyclictest0-21swapper/308:45:133
2721299118,2cyclictest7462-21tr07:35:132
2721299118,2cyclictest6491-21tr08:40:152
2721299118,2cyclictest5797-21taskset08:38:382
2721299118,2cyclictest24127-21cut09:50:132
2721299118,2cyclictest20732-21ssh09:10:022
2721299118,2cyclictest19953-21cut10:45:142
2721299118,2cyclictest19528-21expr08:00:142
2721299118,2cyclictest18324-21taskset09:05:022
2721299118,2cyclictest13005-21ssh10:21:162
2721299117,3cyclictest706-21sed08:25:262
2721299110,11cyclictest0-21swapper/211:35:242
2720999118,2cyclictest6327-21ls10:55:261
2720999118,2cyclictest31982-21wc07:20:001
2720999118,2cyclictest30778-21ssh11:29:551
2720999118,2cyclictest24642-21ssh11:45:151
2720999118,2cyclictest24642-21ssh11:45:151
2720999118,2cyclictest23182-21tail08:05:261
2720999118,2cyclictest22260-21sh12:21:301
2720999118,2cyclictest19670-21taskset10:45:081
2720999118,2cyclictest18119-21perf09:05:001
2720999118,2cyclictest17508-21grep12:00:131
2720999118,2cyclictest16983-21ssh10:43:151
2720999118,2cyclictest15602-21taskset08:58:541
2720999118,2cyclictest12833-21cat07:45:181
2720999118,2cyclictest12778-21sed08:50:261
2720999118,2cyclictest10709-21head10:20:151
2720999117,3cyclictest1875-21ssh11:50:201
2720999110,8cyclictest0-21swapper/109:40:031
2720999110,10cyclictest30033-21cut10:50:221
2720999110,10cyclictest18485-21sh10:25:171
2720799118,2cyclictest870-21tr11:50:140
2720799118,2cyclictest6210-21egrep11:15:160
2720799118,2cyclictest5986-21ssh12:30:550
2720799118,2cyclictest587-21hostname07:20:170
2720799118,2cyclictest5115-21cat07:30:150
2720799118,2cyclictest31631-21ssh09:15:350
2720799118,2cyclictest31078-21ssh12:26:460
2720799118,2cyclictest30500-21ssh10:31:450
2720799118,2cyclictest30396-21cat11:10:140
2720799118,2cyclictest27190-21grep10:10:250
2720799118,2cyclictest25767-21uname11:45:210
2720799118,2cyclictest25767-21uname11:45:200
2720799118,2cyclictest21965-21ssh11:05:160
2720799118,2cyclictest21152-21awk10:45:220
2720799118,2cyclictest16725-21grep09:00:160
2720799118,2cyclictest15439-21ssh10:04:220
2720799118,2cyclictest13533-21ssh11:00:150
2720799118,2cyclictest13331-21munin-plugin-st08:55:000
2720799118,2cyclictest11626-21sshd07:43:430
2720799118,2cyclictest11010-21cat08:50:000
2720799117,3cyclictest11079-21grep11:55:280
2720799110,10cyclictest19145-21ssh09:46:440
2721499107,2cyclictest21733-21grep08:05:153
2721499107,2cyclictest18347-21cat09:05:113
2721299108,1cyclictest3279-21cut07:25:202
2721299108,1cyclictest30667-21cat07:15:182
2721299108,1cyclictest29523-21taskset07:14:552
2721299108,1cyclictest285122sleep208:18:572
2721299108,1cyclictest18824-21chrt07:59:412
2721299108,1cyclictest17366-21cut09:00:242
2721299108,1cyclictest12213-21cat07:45:152
2721299107,2cyclictest23395-21taskset08:07:062
2721299107,2cyclictest13067-21taskset08:51:472
2721299100,9cyclictest4906-21wget07:30:142
2721299100,9cyclictest32253-21ssh09:16:032
2720799108,1cyclictest4208-21expr08:35:130
2720799108,1cyclictest20337-21taskset09:08:230
2720799108,1cyclictest16995-21expr07:55:160
2720799102,6cyclictest29815-21apache_processe07:15:120
272149996,2cyclictest4227-21cstates08:35:143
272129997,1cyclictest106342sleep208:46:002
272129990,4cyclictest225-21systemd-journal08:14:592
272099996,2cyclictest2473-21sed08:30:191
272099996,2cyclictest24034-21wget08:10:141
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional