You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-17 - 18:33

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Wed Dec 17, 2025 12:44:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
275851021240,7sleep04497-21PK-Backend08:35:180
275851021240,7sleep04497-21PK-Backend08:35:180
277283021050,6sleep126-21ksoftirqd/109:15:191
28027862960,5sleep04497-21PK-Backend10:40:260
197399820,72rtkit-daemon0-21swapper/107:09:591
272794999812,38cyclictest2069-21gnome-shell08:45:582
27306952750,19sleep342-21ksoftirqd/307:15:143
197399700,8rtkit-daemon1972-21rtkit-daemon09:42:351
197399700,8rtkit-daemon1972-21rtkit-daemon09:42:341
28363622690,6sleep0161rcuc/012:15:120
28347322690,8chrt4497-21PK-Backend12:10:120
272775026846,14sleep20-21swapper/207:09:372
2727953996725,37cyclictest2826543-21apt-get11:50:103
197399670,8rtkit-daemon1972-21rtkit-daemon08:45:561
197399670,8rtkit-daemon1972-21rtkit-daemon08:36:421
197399670,8rtkit-daemon1972-21rtkit-daemon08:36:421
197399660,8rtkit-daemon1972-21rtkit-daemon08:26:501
272776726545,14sleep30-21swapper/307:09:513
272775826438,20sleep00-21swapper/007:09:450
272794999613,31cyclictest2069-21gnome-shell08:26:582
2727949996121,35cyclictest2733487-21seq07:24:322
197399610,8rtkit-daemon1972-21rtkit-daemon08:58:101
197399610,8rtkit-daemon1972-21rtkit-daemon08:51:101
2727949995924,31cyclictest2728991-21sensors_temp07:10:302
272794599592,15cyclictest2069-21gnome-shell07:16:581
272794199592,53cyclictest2069-21gnome-shell07:20:580
2727949995821,33cyclictest2835657-21seq12:12:382
2727953995719,11cyclictest2832252-21mktemp12:05:023
272795399571,52cyclictest2069-21gnome-shell07:34:583
272794599574,33cyclictest2791666-21cat10:10:131
272795399562,50cyclictest2069-21gnome-shell09:44:583
272795399562,50cyclictest2069-21gnome-shell09:44:583
272794999562,50cyclictest2069-21gnome-shell11:53:082
2727949995622,30cyclictest2804100-21fw_packets10:45:162
2727949995622,29cyclictest2749440-21awk08:10:102
2727949995617,34cyclictest2751623-21cat08:15:182
272794999561,28cyclictest2801133-21/usr/sbin/munin10:35:272
197399550,8rtkit-daemon1972-21rtkit-daemon09:23:591
27846682540,5chrt331rcuc/209:50:102
2727953995415,12cyclictest923-21dbus-daemon12:08:413
272794999542,48cyclictest2069-21gnome-shell07:37:582
2727949995415,33cyclictest2833683-21seq12:05:342
272794999541,30cyclictest2775947-21/usr/sbin/munin09:25:262
272794599545,45cyclictest231irq_work/108:10:581
272794599544,46cyclictest231irq_work/109:08:591
272794599544,46cyclictest231irq_work/107:48:581
272794599544,46cyclictest231irq_work/107:30:581
272795399532,47cyclictest2069-21gnome-shell09:39:583
2727953995322,7cyclictest2792084-21uname10:10:203
272795399532,10cyclictest2069-21gnome-shell09:14:583
2727953995314,13cyclictest2812809-21cat11:10:173
272794999532,10cyclictest2069-21gnome-shell08:43:582
2727949995320,28cyclictest2811584-21timerandwakeup11:05:282
2727949995316,33cyclictest2773304-21cat09:15:282
272794999531,30cyclictest2734429-21cat07:25:212
272794999531,30cyclictest2734429-21cat07:25:212
272794599534,45cyclictest231irq_work/109:35:591
272794599534,45cyclictest231irq_work/109:10:591
272795399522,8cyclictest2839355-21munin-run12:25:023
272795399521,46cyclictest2069-21gnome-shell08:21:583
272795399521,10cyclictest2069-21gnome-shell08:50:583
2727949995224,24cyclictest2839333-21idleruntime-cro12:25:002
2727949995220,28cyclictest2792677-21vmstat10:10:302
272794999521,48cyclictest2069-21gnome-shell08:00:582
2727949995214,33cyclictest2809056-21grep11:00:132
272794599523,45cyclictest231irq_work/108:33:581
272794599522,8cyclictest231irq_work/107:54:591
272794599522,46cyclictest231irq_work/109:00:581
272794599522,46cyclictest231irq_work/108:04:581
272794599521,47cyclictest231irq_work/107:36:581
2727953995117,11cyclictest2789127-21sed10:00:283
272795399511,29cyclictest2796548-21munin-run10:25:013
272795399511,13cyclictest2806652-21taskset10:52:003
272794999515,25cyclictest0-21swapper/209:05:292
272794999514,41cyclictest2818385-21sort11:25:192
272794999512,27cyclictest0-21swapper/211:18:152
2727949995118,29cyclictest2813832-21/usr/sbin/munin11:15:012
2727949995118,28cyclictest2842230-21switchtime12:30:282
2727949995116,30cyclictest2817055-21cat11:20:292
272794999511,44cyclictest438-20systemd-journal07:40:032
2727949995114,32cyclictest2806770-21sshd10:53:322
2727949995113,33cyclictest2823198-21expr11:40:122
2727949995112,34cyclictest0-21swapper/208:20:212
272794599512,8cyclictest231irq_work/108:07:581
272794599512,45cyclictest231irq_work/107:29:591
272794599512,45cyclictest231irq_work/107:29:591
272794599511,8cyclictest231irq_work/109:26:581
272795399502,9cyclictest2728495-21df_inode07:10:163
272795399502,45cyclictest0-21swapper/307:25:583
272795399502,45cyclictest0-21swapper/307:25:583
2727953995018,10cyclictest2837011-21awk12:15:263
272795399501,45cyclictest2069-21gnome-shell09:53:583
272794999503,26cyclictest924-21NetworkManager11:30:162
2727949995017,29cyclictest2821437-21cpuspeed_turbos11:35:152
2727949995017,29cyclictest2821437-21cpuspeed_turbos11:35:142
2727949995017,29cyclictest2768641-21sort09:05:022
2727949995017,29cyclictest2735655-21cpuspeed_turbos07:30:122
2727949995016,29cyclictest2795929-21smartctl10:20:272
2727949995013,33cyclictest2794879-21date10:20:012
272794599508,25cyclictest2802657-21seq10:40:221
272794599502,45cyclictest231irq_work/109:49:581
272794599501,8cyclictest231irq_work/108:16:591
272795399495,23cyclictest2816726-21sort11:20:223
272795399494,42cyclictest0-21swapper/308:15:583
2727953994934,11cyclictest2069-21gnome-shell07:42:583
272795399492,7cyclictest0-21swapper/309:23:593
272795399492,44cyclictest0-21swapper/308:45:583
272795399492,44cyclictest0-21swapper/308:13:583
272795399492,43cyclictest0-21swapper/307:45:583
2727953994914,14cyclictest2826245-40apt-check11:46:583
272794999495,36cyclictest28262892sleep211:47:052
272794999492,42cyclictest2093-21JS10:49:092
272794999491,44cyclictest2069-21gnome-shell09:13:582
2727949994912,33cyclictest2774316-21expr09:20:152
2727949994912,32cyclictest2758664-21cat08:35:212
2727949994912,32cyclictest2758664-21cat08:35:212
2727949994910,34cyclictest2744841-21missed_timers07:55:222
272794599494,27cyclictest2820597-21sed11:30:281
272794599491,45cyclictest231irq_work/108:42:591
272794199492,43cyclictest4497-21PK-Backend07:12:590
272795399482,7cyclictest0-21swapper/308:30:593
272795399482,43cyclictest0-21swapper/309:25:593
272795399482,43cyclictest0-21swapper/309:15:583
272795399482,43cyclictest0-21swapper/308:56:593
272795399482,43cyclictest0-21swapper/308:44:583
272795399482,43cyclictest0-21swapper/308:35:583
272795399482,43cyclictest0-21swapper/308:35:583
272795399482,43cyclictest0-21swapper/307:50:583
272795399482,42cyclictest0-21swapper/309:06:583
272795399482,42cyclictest0-21swapper/308:01:593
272795399482,25cyclictest2839955-21df12:25:153
272795399482,24cyclictest2841724-21hddtemp_smartct12:30:173
272795399482,24cyclictest2815216-21sensors_temp11:15:273
272795399481,11cyclictest2830145-21seq11:57:573
272795399481,10cyclictest2823035-21acpi11:40:093
2727949994818,26cyclictest2840069-21/usr/sbin/munin12:25:202
2727949994818,26cyclictest2765301-21/usr/sbin/munin08:55:282
2727949994817,26cyclictest2836335-21cstates12:15:122
2727949994815,28cyclictest2742707-21cpuspeed_turbos07:50:122
272794999481,42cyclictest2828835-21/usr/sbin/munin11:55:172
2727949994814,29cyclictest2798514-21awk10:30:102
2727949994814,29cyclictest2786524-21cpuspeed_turbos09:55:132
2727949994811,33cyclictest2788242-21cpuspeed_turbos10:00:122
2727949994810,33cyclictest2730395-21cpu07:15:112
2727945994839,6cyclictest231irq_work/107:14:581
272794599483,21cyclictest2841771-21cat12:30:181
272795399472,7cyclictest2804291-21memory10:45:203
272795399472,42cyclictest0-21swapper/309:49:593
272795399472,42cyclictest0-21swapper/309:01:583
272795399472,42cyclictest0-21swapper/308:29:583
272795399472,42cyclictest0-21swapper/307:35:593
272795399472,41cyclictest4497-21PK-Backend07:20:583
2727953994721,8cyclictest2799267-21ntp_states10:30:253
2727953994720,7cyclictest2822793-21/usr/sbin/munin11:40:003
2727953994720,7cyclictest2822793-21/usr/sbin/munin11:40:003
272795399471,42cyclictest0-21swapper/309:32:583
2727953994713,8cyclictest0-21swapper/308:05:133
2727949994718,25cyclictest2740768-21/usr/sbin/munin07:45:212
2727949994715,28cyclictest2748600-21switchtime08:05:282
2727949994714,28cyclictest2802144-21cpuspeed_turbos10:40:142
272794999471,41cyclictest2781885-21cut09:40:212
272794999471,41cyclictest2781885-21cut09:40:212
2727949994712,30cyclictest2778286-21sed09:30:212
2727949994710,32cyclictest2843638-21memory12:35:202
272794599473,26cyclictest2838601-21ntp_states12:20:221
272794199471,42cyclictest4497-21PK-Backend09:27:580
272794199471,42cyclictest4497-21PK-Backend08:24:580
272794199471,42cyclictest4497-21PK-Backend07:18:580
272795399462,24cyclictest2835139-21ntp_states12:10:223
272795399462,24cyclictest2794819-21/usr/sbin/munin10:20:013
272795399461,24cyclictest2796913-21grep10:25:113
272795399461,24cyclictest2796913-21grep10:25:113
272795399461,23cyclictest2811374-21ntp_states11:05:243
2727953994611,8cyclictest0-21swapper/311:30:133
272794999469,33cyclictest2756700-21expr08:30:152
2727949994638,5cyclictest0-21swapper/208:51:582
2727949994615,27cyclictest2790071-21cstates10:05:132
272794999461,40cyclictest2808112-21cat10:55:292
272794999461,40cyclictest2779659-21expr09:35:122
2727949994610,31cyclictest2783684-21sh09:45:232
272794599466,18cyclictest1137-21runrttasks11:05:021
2727941994639,4cyclictest4497-21PK-Backend09:34:580
28361052450,15chrt26-21ksoftirqd/112:15:081
272795399452,7cyclictest2802345-21df10:40:153
272795399452,24cyclictest2801743-21munin-run10:40:013
272794599455,17cyclictest2796528-21munin-run10:25:001
272794599444,19cyclictest2787065-21head09:55:211
2727941994437,4cyclictest4497-21PK-Backend09:46:590
2727941994437,4cyclictest4497-21PK-Backend08:59:580
2727945994336,4cyclictest0-21swapper/107:59:581
272794599433,14cyclictest2817924-21cat11:25:121
272794599432,20cyclictest2815418-21timerwakeupswit11:15:301
272794999421,37cyclictest2797088-21expr10:25:112
272794999421,37cyclictest2797088-21expr10:25:112
272794599424,15cyclictest2841218-21sort12:30:011
272794599424,15cyclictest2798259-21munin-run10:30:001
272794599424,15cyclictest2798259-21munin-run10:29:591
272794599424,15cyclictest2788736-21sed10:00:211
272794599422,18cyclictest2827608-21sed11:50:181
2727941994214,6cyclictest141rcu_preempt12:35:000
197399420,35rtkit-daemon13-21ksoftirqd/009:05:320
272794599414,15cyclictest2809626-21awk11:00:231
272794599414,14cyclictest2785543-21smartctl09:50:261
272794599413,16cyclictest2844533-21perf12:40:001
272794599411,35cyclictest26-21ksoftirqd/107:20:161
2727941994134,4cyclictest4497-21PK-Backend07:35:590
272794599407,29cyclictest2817580-21cron11:25:001
272794599407,10cyclictest2778620-21smartctl09:30:271
272794599406,9cyclictest0-21swapper/112:07:591
272794599403,14cyclictest2789835-21/usr/sbin/munin10:05:191
272794599401,16cyclictest2804441-21ntpq10:45:221
272794599401,15cyclictest2807147-21/usr/sbin/munin10:55:111
28221992390,6chrt0-21swapper/111:35:281
28221992390,6chrt0-21swapper/111:35:281
28119662390,8chrt4497-21PK-Backend11:08:350
2727953993917,7cyclictest1137-21runrttasks12:39:423
272794599396,12cyclictest0-21swapper/112:15:011
272794599394,13cyclictest2793296-21/usr/sbin/munin10:15:171
2727941993926,8cyclictest438-20systemd-journal09:04:550
28401792380,7sleep04497-21PK-Backend12:25:220
272794599382,14cyclictest0-21swapper/111:40:281
272794199381,15cyclictest4497-21PK-Backend08:01:400
272794599379,23cyclictest2753559-21tail08:20:221
272794599377,8cyclictest0-21swapper/110:37:561
2727941993730,4cyclictest4497-21PK-Backend07:29:580
2727941993730,4cyclictest4497-21PK-Backend07:29:580
2727953993623,9cyclictest42-21ksoftirqd/307:55:243
2727953993621,11cyclictest923-21dbus-daemon11:30:013
272794599361,12cyclictest2813877-21apt-config11:15:011
2727941993629,4cyclictest4497-21PK-Backend09:24:590
2727941993625,7cyclictest4497-21PK-Backend08:48:580
2727941993625,6cyclictest4497-21PK-Backend07:50:580
27737732350,5sleep0161rcuc/009:20:020
272795399352,29cyclictest2790609-21ntp_states10:05:243
2727953993519,11cyclictest2808792-21grep11:00:013
272794599357,24cyclictest2798672-21tr10:30:131
272794199359,5cyclictest141rcu_preempt11:15:000
2727941993528,4cyclictest4497-21PK-Backend09:43:590
2727941993528,4cyclictest4497-21PK-Backend09:43:580
2727941993521,10cyclictest13-21ksoftirqd/008:40:530
272794199351,30cyclictest4497-21PK-Backend11:32:420
272794599344,8cyclictest0-21swapper/111:45:191
2727945993420,10cyclictest2740597-21sed07:45:011
272794199347,13cyclictest141rcu_preempt07:40:130
2727941993427,4cyclictest4497-21PK-Backend08:52:590
2727941993424,6cyclictest2822788-21turbostat11:39:590
2727941993424,6cyclictest2822788-21turbostat11:39:590
2727941993418,12cyclictest2818636-21awk11:25:250
2727941993418,11cyclictest2839321-21sh12:25:000
2727953993318,11cyclictest2810342-21sshd11:04:573
272794599333,11cyclictest26-21ksoftirqd/112:02:021
272794599332,9cyclictest0-21swapper/110:50:421
2727941993326,4cyclictest4497-21PK-Backend11:54:310
2727941993323,6cyclictest4497-21PK-Backend07:59:590
2727941993322,6cyclictest4497-21PK-Backend08:08:580
272794199331,9cyclictest4497-21PK-Backend08:25:220
2727941993319,10cyclictest2793192-21cat10:15:020
2727941993316,13cyclictest2795776-21ntpq10:20:220
272794199331,4cyclictest4497-21PK-Backend10:30:470
272794199331,28cyclictest4497-21PK-Backend10:25:310
272794199331,28cyclictest4497-21PK-Backend10:25:300
27498222320,5sleep04497-21PK-Backend08:10:150
2727953993215,12cyclictest2787150-21ntpq09:55:233
272794199322,11cyclictest2801718-21turbostat10:40:000
2727941993217,10cyclictest2805588-21tr10:50:120
2727941993216,11cyclictest438-20systemd-journal11:24:550
272794199321,4cyclictest4497-21PK-Backend07:34:280
272794199321,27cyclictest0-21swapper/011:02:360
272794199321,17cyclictest27940372/usr/sbin/munin10:15:230
272794599312,8cyclictest0-21swapper/111:55:251
2727941993121,6cyclictest4497-21PK-Backend08:30:590
272794199311,8cyclictest4497-21PK-Backend12:05:260
2727941993117,9cyclictest2824823-21date11:45:010
272794199311,27cyclictest4497-21PK-Backend08:18:320
2727941993016,10cyclictest2844042-21grep12:35:280
272794199301,4cyclictest4497-21PK-Backend10:57:400
272794199301,4cyclictest4497-21PK-Backend09:57:000
272794199301,4cyclictest4497-21PK-Backend09:36:060
272794199301,4cyclictest4497-21PK-Backend07:45:590
272794199301,25cyclictest4497-21PK-Backend11:49:520
272794199301,24cyclictest4497-21PK-Backend11:58:030
272794199301,14cyclictest4497-21PK-Backend11:15:160
272794199291,5cyclictest4497-21PK-Backend10:07:320
272794199291,5cyclictest4497-21PK-Backend09:51:070
272794199291,10cyclictest4497-21PK-Backend12:03:430
2727941992821,4cyclictest4497-21PK-Backend10:46:460
2727941992819,5cyclictest4497-21PK-Backend09:10:270
272794199281,10cyclictest4497-21PK-Backend10:01:000
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional